Contact module roadmap: 80% usecase of webforms in core

This is so exciting +1 to the idea.

The related blocks us on proper handling of submissions

Also there's could be some migrate templates to convert data from d6/d7 (web|entity)form into basic forms and massages

I think contact storage is up to date with most all of changes in core and creating a patch will not be an issue.

The only blocker issue for pluggable storage is #2629540: Mismatched entity and/or field definitions after installing module

I would love to help into this to push this functionality into core.

Retitling to make the "why" of this plan obvious. :)

@larowlan and myself talked about it one day and we realized that at some point "renaming" the contact module to something like form would be a big help for users to realize how flexible the contact module actually is in Drupal 8. Maybe we could add it to the agenda.

Might I suggest adding this to the roadmap: #2720283: add plugins to Contact form types, remove special-casing for 'personal' contact form type?

How much of phase 2 could be done in an experimental module (i.e. contact_storage for example)?

Also would it be worth making that new experimental module called 'forms'? We could then move some things out of contact module, or even deprecate it.

As per our(@larowlan, @berdir, @pameeela and me) discussion with @webchick and @xjm, all these things will only go into feature branches and in contact module. We removed all those issues/features from the roadmap which can't be included to the contact module i.e. HTML mails, per contact form permissions and e.t.c.

The idea is to keep maturing and adding those features in contrib i.e. contact_storage and if we think after user testings that we need some of these features in core then we'll consider adding a new experimental module to core.

IMHO, an experimental new module doesn't make too much sense. Most of those features aren't really experimental, they're already being tested in contact_storage and they're AFAIK all opt-in.

Actually moving any relevant code like the entities and all code around them would lead to API changes or duplication of those entities. One thing we could consider is renaming contact module to forms module just in the UI, like we did with Simpletest, although that has downsides too.

My suggestion would be to add more features to contact.module now and then just rename it in 9.x to forms.

Forgot the most important part: An experimental module gives warnings on the status page and we essentially tell non-experienced users to not to use it. That's IMHO not the message we want to send.

I like the idea of a rename to forms, but I don't think we can do it until Drupal 9 either.

Well, if we are super radical we could copy contact module to forum module and call contact module as deprecated as of now, and just work on the new forms module.
Not sure this is a good idea though, because, at least at that point in time, it would require new patches for all existing issues.

Yeah I was thinking of something like @dawehner's #29. On top of that we could probably have contact module depend on forms module and subclass bits to minimize duplication.

+1 on #29 and #30

I like idea of Forms but once we fork experimental module better to reaseach ymlform module ideas as well, there's ui builder working

I think forms approach needs design to decide what is 80% of use cases

Contact storage needs UI to construct, list and expose
The main issue in this are suck for years #1997692: Create contact form block

Another +1 to #29 and #30

Looking at this as an end-user, even just changing to Forms from Contact form would be a major improvement. It just feels wrong to "Add contact form" when you're not working on a contact form.

The UI is contact-specific in a few ways, including the "Selected" column, which is confusing. I think it is a blocker to adoption of contact as webform's successor, not just because of the name/UI, but things like every form has a subject and message field that you have to hide on display. It feels like a workaround rather than a solution.

We discussed in the call that we would spin up a demo for some client-types to test but tbh I don't think it's ready for that currently.

Yeah, renaming in the UI probably makes sense.

But I still don't fully get how we'd benefit from duplicating the contact.module into a new form.module?

We'd have to do it as the very first thing, as a result, we'd have two identical modules, one deprecated and one experimental (?). Only then we could start adding new features one by one. The alternative would be to develop the forms module as a contrib module first, possibly based on a merge of contact + contact storage and only get it into core when we consider it complete (enough). I suppose it *would* be a good match for the development model proposed by Dries in Amsterdam but so far, we haven't done that.

Also, switching from contact to forms module for existing sites sounds pretty painful. They'd have to re-create or migrate all configured forms and data if they want to keep that. And existing contrib modules like contact_storage would no longer work, we do not plan to move every feature it has into core, e.g. HTML mails/swiftmailer integration. Keeping contact_storage working against a 8.x-2.x moving target would of course also be a challenge. We also have a ton of custom code integrating with contact forms, I suppose other sites as well that would have to be rewritten if they want to use the new module.

As I wrote above, none of the phase 2 features nor most of the phase3 things seem to be an issue for BC and adding them in smaller steps would result in immediate improvements for existing and new sites. Some challenges would exist either way (configurable storage, I suppose that simply means always create storage, and then save or not. otherwise views integration and so on would be really fun). One that possibly only exists with a non-experimental module is to figure out how to keep views configurations up to date without breaking customizations. But that's a problem that contrib already has to fight with and having better support for that in core would be a win for everyone (#2684081: Modules need a way to keep their configuration up to date).

A bit of feedback on the specific steps in phase 3, 2 seems fine:
7: Still not a fan of the menu suggestion for the category regression/use case. I think the options email field type that we have in contact_storage now is a much better replacement for that. As mentioned before, once we have that in core, I'd even go as far as suggesting to migrate 6.x/7.x contact forms into a single contact form with a category field.
9 seems to be a duplicate of 6
10/2 are again related and being worked on in #2734289: Configurable mail subject with token support (Actually, plenty of the phase 3 features have issues/patches in contact storage already)

I don't think we have to do that rename now, but it's something that doesn't have to wait until 9.x either - we could do it towards the end of the 8.x cycle for example.

Before assessing the technical challenges, let alone starting to write patches, I think we need to decide whether this is actually supported as a core initiative.

My 2c is that we need to completely remove all of the contact assumptions, and on some levels that might be easier starting from scratch that retrofitting what we have. Otherwise, it's going to be a half-baked solution. So maybe this belongs in contrib (or at least a feature branch) for the time being?

I created a few more issues to this end and discussed with @larowlan but the piecemeal approach seems daunting the longer I spend looking at what needs to be done. It will be hard to get the UX right by undoing one thing at a time.

I would suggest to schedule a call/discussion with the product maintainers and potentially release managers. We need to have a clear understanding of what we want to achieve, the exact "what we are gonna name it" is it "experimental or not" should be secondary to that discussion.

Make a Forms Module, moving all the base code from Contacts module into Forms Module. Have Contacts Module utilize Forms modules APIs to continue to work without duplicating code. We then leave Contact Module the way it is as a layer on top of Forms Module. At this point we can put all our focus on Forms Module and built it as a true WebForms replacement.

Should at any time the API of Forms change in a way that breaks Contacts Module we then make a patch to Contacts Module to fix the problem. Contacts Module will no longer have any real underlining APIs of its own and will be 100% reliant on Forms Module. At some point we can start to phase out contact Module as Forms module takes shape.

Forms Module should strive to meet 100% of what WebForms can do with the additions of all popular add-on modules for WebForms. This will give a powerful Forms API that other modules can work off and build on.

Forms Module should have a simple API for building forms but also have a drag-n-drop form builder UI. Forms should be easy to assign to menus, blocks, users and roles. There should be full integration with the Rules Module for complex form logic.

Storage, redirect, and external mail services should be built into Forms Module.

Good points in #41, yamlform already has nice form builder, I'd like to get comments from maintainer here

Hi,

I am the maintainer of the YAML form module, which has gradually turned into a somewhat D8 port of Webforms. I am 100% on board with the goal that the Contact Storage module (aka an Entity/Field based form builder) can address 80% of the form building use cases for D8/D9. My module is only trying to address the other 20%.

As I finish up my module, I would like to start helping out with this initiative. For now, I want to share my experience researching and building a form builder for Drupal 8.

The most successful and popular form builders (Wufoo, Google Forms, etc...) are generally very simple to use. There use case is basically, build a form, collect results, and download them. The general challenge with using Drupal's Entity/Field system as a form builder is that it is too robust (and complex). I feel for a form builder in core to be successful it needs to feel simple and hide Drupal's robust Field API.

For example, Drupal's Field API requires multiple steps to create a form input (aka a field). Ideally this should be streamlined into a single form with reasonable default values. The default form display, submission display, and email display could be part of an input's main single form. Ideally, there should also be one single UI for organizing the form and submission display.

Also, my form builder started out by sending plain text email and then I added support for HTML emails. I realized that this was the wrong approach, because almost everyone can view HTML email and most people prefer HTML emails. Drupal's form builder should focus on building the HTML email first and then just downgrade the HTML to plain text. BTW, Drupal core really should support sending HTML email out of the box. One challenge is that the Field API templates that ship with core are not exactly HTML email friendly.

I feel like I am just touching the tip of the iceberg when it comes to the challenges of implementing a form builder for core. I have to agree with @Patrick_McMahon the best approach is probably to start with new module that focuses on the form build UI first, submission handling and viewing second, and finally sending emails. At the same time, I think Contact Storage module should move ahead with an MVP (Minimum Viable Product) so that the community has something to use, which then can easily be migrated to the new form builder in core.

Finally, if I got a little more help and general code review with my YAML form module, it could also help calm the grumblings about Webforms not being available for D8.

Hope this helps. ~jake

So I actually think that the name bike shedding was quite useful in shaping my thinking on this. It spawned a lot of conversation about forms in general and how they can be improved in Drupal core across the board. These ideas are great and I hope that we can progress toward a more intuitive UI for all forms.

But for me it highlighted the disconnect between what Field UI does and what the average editor wants/needs. The main distinction is, with contact forms, the submission is the entity. With Webform, the form is the entity. I believe that we need a "form is the entity" solution, not necessarily in core, but somewhere in the ecosystem.

Until now that has been Webform, and Webform is awesome. I can understand the many reasons that it hasn't been ported yet, and the many reasons to try to use Field UI as the solution rather than building a separate one. But it is very telling that there is already a Webform-style module in D8 in YAML form, which is a different approach to the same problem. YAML form is also awesome and I'm really impressed with how advanced the functionality is already. The fact that this module exists and is stable and actively supported is evidence that we need this functionality.

There is already some great work on Field UI with Dream Fields, and to make Contact more generic. But I can't really see a point where it supersedes the Webform/YAML form functionality. Most people who want to make a form just want to make a form, and creating a whole entity type for that is overkill.

Of course I could be wrong about that. Even so the point where the Field UI form stuff can replace the simple(ish) form builder is probably still pretty far off and in the meantime we need a YAML form to fill the gap. So thank you @jrockowitz for providing it!

Without trying to downplay yamlform (which is frankly a phenomenal effort and wonderful solution to this problem), the developer in me that sees the huge overlapping functional and technical challenges of a full-featured form builder/submission manager and entity/field UI can't really reconcile that these are two different problem spaces. If the amount of effort that went into solving problems twice (the entity way and the webform way) went into improving making entity API less daunting, then all of Drupal would benefit, not just the form-builder use case.

Yamlform is awesome, but it doesn't help shop owners add attributes to their commerce products. The conditional functionality awesome, but it doesn't help clean up /node/add pages for complex content types.

I realise it's asking a lot, but I feel this initiative is the "right" way , forward thinking and the D8 way, free from special snowflake pockets of code in walled garden contrib modules.

@Sam152 My goal with the YAML form module is to provide the Drupal community with an Open Source customizable equivalent to a product like SurveyMonkey and Wufoo. Webform and YAML form are contrib modules that are products. Drupal core is a framework that helps build products. Drupal core needs to improve its form building framework and hopefully this will eliminate the need for a form builder in contrib. At the same time, I think that Drupal's adoption has benefited from products like webforms and also distributions.

You are right the YAML form module "doesn't help clean up /node/add pages for complex content types". I feel things like this simple task and important feature needs to be addressed in Drupal core. Maybe there needs to be an 'Entity customization' module that allows any entity form's layout, submit buttons, confirmation action, etc… to be altered using a UI.

A better example, is entities should have a global reusable way to trigger notifications (via the Rules module) that can be sent via email (and even SMS). Right now, both the Contact module and the User module are sending custom email notifications when new entities are created and updated. If a unified and pluggable mechanism to send notifications was added to core it would provide the confirmation functionality required by a form/survey builder.

The exporting of submission data could be addressed by a generic entity export module, that could just live in contrib, which would also be extremely useful for managing other entities like user, comments, and redirects.

A super powerful concept would be the automatic generation of the initial manage submissions view for new forms (aka bundles). My first d8 migration actually generated admin and teaser views/displays for all 30+ content types on the website.

It is also very likely that the handling of form submission drafts and previews could be addressed by the workflow initiative. Also the workflow initiative would benefit from a global notification/messaging API

Everyone is agreeing that there needs to be an initiative to improve Drupal's form and survey building capabilities. We now need to figure out the best approach.

I am suggesting smaller sub initiatives that address discreet pieces of reusable form builder related functionality. Ideally the Forms module could just contains a form config entity and a submission content entity. that leverages cores APIs to create a form/survey building UX

So far the possible new functionality/modules/initiatives I have potentially outlined are …

Entity customization: Allows site builders to customize an entity's form and confirmation handling.

Entity notification: Allows site builders to add notifications to an entity's CRUD operations.

Entity export: A simple UI that allows any entity and fields to be exported (as a CSV).

Entity Views: Automatically generates admin views for new content entity bundles.

...and Rules probably needs to be in core
…and core should support sending HTML email.

Above are just my initial thoughts. I also think Ted Bowman's Entity form/Eform module is another good starting point because he has had to address most of these requirements.

@Sam152 Let's say you're right that this is the "right" way long term, but if we need a form builder now, contact does not suffice. So what do you propose in the meantime, if not YAML form or something similar?

Very interesting points on this thread.

One of the things that has always saddened me with Drupal community is how often there are duplication of efforts that can divide our collective contributions. (Of course I get that can also inspire innovation too).

At this point my vote would be to start with YAMLForms and grow from there. As of right now YAMLForms is the most comprehensive and closest thing to webform out there. With D8 adoption rate increasing, more and more site builders are going to reach for the webform replacement and stumble upon YAMLForms. If you look as the YAMLForms downloads statistics you can see the spike in just the past few months.

The more the YAMLForms maintainer can get help from the community the better this solution can be and maybe even inspire the creation of a better API like mentioned earlier.

I heard keynote from Josh Koenig from Pantheon talk about how WordPress works a bit differently where a featured doesn't' get added unless it helps the blogger. I think this is an interesting lesson that the Drupal Community could learn from. How can developers help the site builders build sites?

I totally agree that we need to define actionable items and reorder priorities on current state of contact module issues

As of now I though plan as
- storage (any form submission needs storage with current core's /dev/mail)
- html mails (suppose improvements in decoupling mail, book module already have render of print version) #2223967: Do not decode a contact message twice
new topics
- form builder
-- that could cooperate with layout plugin issue #2296423: Implement layout plugin type in core and with #1787942: Allow assigning layouts to pages to make contact pages reusable for simple contact landing pages
-- could collaborate with https://www.drupal.org/project/dream_fields to improve field UI
- submission logic customizations

So I'm focused on

8.2.0 - decide on mail issues and extensions like #2689265: Allow changing the 'Send message' contact form string and optionally hiding the preview button also I'd say we can move storage from tests to module
8.3.0 - builder/layout - experimental module
... tbd

Finally we can polish "Message" entity to incorporate sort of https://www.drupal.org/project/collect

I am suggesting smaller sub initiatives that address discreet pieces of reusable form builder related functionality. Ideally the Forms module could just contains a form config entity and a submission content entity. that leverages cores APIs to create a form/survey building UX

Can you please elaborate your's opinion on order and steps for core to try achieve in minor versions of core?

Technically speaking I'd simple implemented following

- polished storage plugability
- decoupled submission
-- introduced submission event (mail subscriber is core default implementation)
-- Introduced Submission API
- finished api of form building
- forms api somehow... builder should create some entity that implements FormSubmissionInterface

Hi there! This is an issue that we'd ideally like to move to the new "Drupal core ideas" queue when/if #2785735: [policy, no patch] Agile process for bigger changes in core (Part A: Ideation) goes through (hoping for the next week or two). If you could read that issue and provide your thoughts on that, that would be great!

Doing that. :)

@phase 3 issue 5. Path integration to allow simple alias management of contact categories

Why not integrate contact form into pathauto so that i can generate custom paths automatically?

@MartinMa
Well, pathauto allows you to generate path aliases automatically. This particular issue is about being able to set a custom path, which is the opposite of automatic :)

Pathauto does only support content entities at the moment and changing that will not be trivial.

Contact storage adds a alias field to contact forms now, and it would be pretty easy to move it to contact.module in core. Just like many of the other features it has.

The formal approval issue #2807777: Formal approval for Phase 2 of 80% usecase of webforms in core.

I have just been talking with the maintainer of eForm which it looks like this is going to replace.

@andypost the eForm module pretty much does what you mention. It has a config entity and a content entity and then some other "formy" bits.

I have had to create quite a few forms for my employer and as such I have created a bunch of modules (in sandbox and only local at present - haven't written any tests yet) to make the form building process a lot nicer. Mainly make it quite easy to create forms from the manage form display (mfd) page including adding fields (using presets to make it a one page affair), edit/delete from the (mfd), controlled fields (basic conditional fields for radios/checkboxes), draft support even with required fields etc.

I really really like the idea of renaming this to something like form module and then contact could just be an addon/sub module that leverages this. Just point me at the form (groan) to sign up at. Happy to help where ever needed.

for "Needs issue summary update"

I think this (and all the related issues) can be closed.

No activity in five years says a lot and I think it’s because webform has since become the obvious choice, so this is just not needed.

Yes but core already has contact storage module in tests)

I can't get why it stuck on approval, but it could use to update summary pointing to contact storage contrib and bringing it into core

Lots of my peers using contact module instead of too-much-features of webform, storage is not needed most of time

I think it's ok to ping @webchick about vision

btw Interesting to explore to store messages in keyValueExpirable storage instead of full blown entities

Yeah I didn't mean development activity, just further discussion. I would expect if this was something with broad support or served a common need, we'd see at least some +1's in the meantime.

I personally don't use contact/contact storage ever and the reason that we pushed for this at the time was that webform didn't exist for D8 yet.

So as a high-level idea I just don't think that '80% usecase of webforms in core' is needed anymore. Contact and contact storage solve the simple use cases and webform gives you all the bells and whistles.

Same as #2807777: Formal approval for Phase 2 of 80% usecase of webforms in core -- this lost steam once Webform for D8 came out.

There may be a case for reviving something like this for Starshot but I think that we are better off starting fresh as there is a lot of outdated info here.

this lost steam once Webform for D8 came out.

Hm, I think this has been closed a little bit too early (no offense) and I rather would say that issues in general "loose steam" sometimes (even for years) especially after long discussions before activity comes back with new insides or motivations. It is not an indicator for a general lack of interest. As one of those who was involved in the link module into core initiative I would like to recommend not to "kill" the conversation about such things too early. I think the title of this issue is not appropiate regarding the motivation of people I have seen lately to vote for extending the contact module. Maybe this is the reason for missing +1s here.

"80% of Webform" in the title is far too much to start a discussion about extending contact to get community momentum working on it. 80% of Webform is very much! And why using Webform as the point of departure when it is actually about to extend contact it's functionality a little bit to prevent people installing complete large module suites just for some parts of it needed. I think I know why it has been reasonably titled like this initially. But I think we should take another look on it today from other angles. Especially since core has extended so much since day one of this issue that many things combined in core "could" actually make it already almost (despite of some missing parts).

Another voice from here including co-workers and clients that Webform is too overblown in many not form centric scenarios. Especially regarding all the libraries and info popups in the system and the "feeling" it creates for UI admins, seeing another system inside the system. Which it is far too much for more than 70% of most use cases where people actually just wanted one thing: a fieldable contact form (already in core) and for some bigger projects maybe another simple storage possibility (like with comments) with the option to (bulk) delete them. I would bet this is more than 70% the reason for the use count of Webform in the last 2-3 years. Which would mean that less of 10% of Webform's features are used in many many of its installs. The rest is possible with core today anyways (statistics, views, filters, etc.).

If we need to keep this title for historical discussion reasons maybe we need another issue to start this discussion from scratch (maybe not by starting from Webform). I am sure there are at least 5-6 core contributors interested to write up some thoughts, which I have spoken with about this issue in the last years. I'll put it temporarely to "Need review" for some more thoughts again, if it is OK for you all. Feel free to close again if we agree that it needs to be revamped in another issue.

Thanks for posting and your interest in improving this!

"80% of Webform" in the title is far too much to start a discussion about extending contact to get community momentum working on it.

Exactly, which is why this issue is closed. At the time of this proposal, there was no Webform for D8, so the scope of this was to fill a gap that no longer exists.

I don't think anyone would object to getting momentum on contact, but as you suggested, that is better done in a new issue with up-to-date information and a more realistic goal.

FWIW I agree with you about webform being too complex for may use cases, but for me the better option would be a simple version of webform rather than trying to make contact work, which also has many complexities, including that it's already in core.

Possibly of interest to you is the Contact form recipe work track for Drupal CMS.

Thanks for coming back on this so quick. I agree with that this issue can be closed (sorry for the noise) since I've found already more or better suited issues already covering my points in the time I tried to think about creating an issue for this thoughts I thought which need to be layed out. What indicates that my assumptions are not missing the community interest in extending Drupal core contact module. So we all can agree the underlying topic is fully covered in other issues and there is nothing left I need to worry about to be missed here or elsewhere for those people. Maybe I was too quick here while I humbled over this issue. BTW, I wonder why it popped up in my dashboard.