Views integration

Moving to Webform queue, see #2827845: [roadmap] YAML Form 8.x-1.x to Webform 8.x-5.x.

Hey, gentlemen!

I got some code in regards to webforms integration with Views. I am still definitely far away from anything that can be called "stable", but I prefer to share it as soon as possible while keeping ongoing work & improvements.

The patch I am attaching has the following features:

• All fields in the webform can be individually added into the view (i.e. you can create a View with table display and specify which fields from webform submissions should be present in that table).
• All fields (to the best of my limited knowledge, hehe) can be sorted. As it's stated in the issue description, the values are stored in mediumtext column, which is not really performant, but ... I guess that's the best you can get.
• All fields (again, to the best of my limited knowledge) are clicksortable.
• All fields are filterable: basically the filter will present you the same form element as during form submission and then the value submitted in the filter is compared against the existing form submission entries.

What I've tested:

• text field & select form elements on fields, sorting, filtering (normal & exposed)
• works nice when you use contextual filter to make sure your view displays submissions of only one specific webform (I think the majority of use cases will be exactly this way - to display a table of submissions for a specific form... just like it's pointed out in the description of this issue).

What I am planning on doing:

• test with more elements
• double check access rights (if the current user has no "view" access on a webform submission entity, then this webform submission entity should be not present in the resultset)
• revisit DB indexes - at some point, when the code and approaches it uses is a bit maturer and stabler it is a very good idea to double check existing indexes in the DB on webform submission tables since sorting/filtering on mediumtext column promises to be a great pain :)
• I should also explore whether it's possible to expose all this views integration through some kind of "views handler class" in entity type of webform_submission or I really have to do it via hook_views_data() & hook_views_data_alter()

After all, the current patch while is still tested very poorly, yet already ships some functionality that may be useful for end users of webform module.

Hi! :) Thanks for your feedback! Your participation at least on the level of bird's-eye view strategic planning will be very helpful, since I've next to none past experience of working with webforms, so I might not be fully aware of the most common use cases and some architectural visioning you've put into the module.

Yes, I'll enrich the code with comments, yesterday I just published a really dirty version of it. I've put up the code myself. Basically I studied the DB structure of {webform_submission} and {webform_submission_data} tables and reviewed the existing plugin types (WebformElement, WebformExport, WebformHandler) and came up with the code for views integration.

The only kind of "anti-pattern" I recognize is that eventually I did not expose {webform_submission_data} table to views through describing it to views in hook_views_data as a table that can be joined on {webform_submission}. I did not do so, because such exposure implies one-to-one relation and we are actually having one-to-many relation between {webform_submission} and {webform_submission_data} - we have to join {webform_submission_data} for each field that will participate in the view. That's why I did not expose the {webform_submission_data} table to views but instead use Drupal\webform\Plugin\views\WebformSubmissionViewsHandlerTrait trait to join {webform_submission_data} whenever any views handler needs another join of it.

Fields

Fields are done in a pretty stupid yet beautiful way: after running the views query there is WebformSubmission entity available, so I just invoke normal rendering procedure on it to get a field displayed in the view result. That means {webform_submission_data} is not JOINed.

Sorting & Filtering

Sorting and filtering does require joining {webform_submission_data}. I left join it (I prefer LEFT over INNER just to cover the case when there's no such field attached to webform or the webform submission has no value for this field). Once joined, I simply do necessary WHERE or ORDER BY on the value column.

Some questions

Would you prefer to keep the views integration within webform.module or to create a separate additional webform_views.module where all views-related integrations will be kept?

You said that listing all webform fields as views fields will be too much. I think you're right, but do you have any idea on how this can be overcome? Maybe just like you have "enable" checkboxes to enable only specific form elements on the webform settings page, maybe we should also have "enable views integration" checkboxes right there? I speak about /admin/structure/webform/settings settings form.

I just discovered that there is also the notion of "delta" in the webform submission values. Hehe, whouuuuups :) How are we supposed to sort in this case? Basically, if I have a webform with "checkboxes" element and there is a submission with 2 checkboxes ticked on within that field. This means my submission will have 2 deltas for that webform element, how do I sort in this scenario? The only thing that comes to my mind right now is to sort on the value where delta=1. Again, that's where you strategic guidance will be helpful, because I am poorly knowledged about common use cases.

There is also the notion of pages in the webform. Do you think that should be somehow accounted in the views integration? At the moment I do not have any idea about how this "page" notion can be applied within views but if you have any idea, do share it.

With your permission I'll create some kind of table in the main description of this ticket where we'll keep track of which elements are covered to which extend. Something like:

Something like the table above, it's a very visual and easy way to grasp where we stand and probably will be useful for folks who come here for quickly applying patches and using it in their projects, since they'll quickly see whether the current progress covers the elements they need.

Yes, indeed I haven't realized field tables are something really similar to {webform_submission_data}. Now, after peaking into views_field_default_views_data() function I was able to get rid of my anti-patter and now {webform_submission_data} is actually JOINed through standard views procedure.

I've one more question about architecture of webform module: there is {webform_submission_data}.property column. What purpose does it serve for? In my all my dummy test webform set ups it is always empty and the existing comment in the DB did not really explain to me its meaning (probably because I've little knowledge about webforms in general).

I am also starting to have some ideas about good architectural solutions to this views integration. Basically, it's evident that putting everything into hook_views_data() is ugly approach, since each webform element may have lots of its own logic and context. So ideally views integration should come from WebFormElement plugins and then we'll only have some thin logic in hook_views_data() that connects what WebFormElement has to say with existing webforms setup (what webforms with what fields are created and how all of it maps into views terminology).

So I would propose to introduce additional interfaces... either single interface WebformElementViewsInterface that probably should extend WebformElementInterface and will determine what views-related data webform element plugins must report about themselves... or we could do like 4 interfaces:

1. WebformElementViewsFieldInterface
2. WebformElementViewsSortInterface
3. WebformElementViewsFilterInterface
4. WebformElementViewsRelationshipInterface (in case we will also figure some way to do relationships)

... Or there is 3rd alternative, we could actually add necessary method(-s) related to views integration into WebformElementInterface and then simply implement it with empty views integration (which is a reasonable backward-compatible default) in WebformElementBase.

And then each webform element plugin type will implement the interface(-s). Again, it's pretty similar to how core Fields are "cooked": each field reports its views data via hook_field_views_data() and then this hook is invoked in views_views_data() where the "per-field-type" views data is expanded onto the existing field set up (which fields exist on which entities). I propose to follow the same path.

I am also cracking right now a rather complicated problem. I envision that majority of use cases for such views integration will be to have a path webform/{webform}/view-submissions where you would display a table of existing submissions (and this table will be backed by a view). But here is one problem: you want to have different views for different webforms, as you were pointing out, in the future we will even like to spawn a new view for a new webform out from some kind of template view. The problem is that all of these views must be attached to the same route. My current solution is not to use webform/{webform}/view-submissions but use specific ones webform/webform-id-1/view-submissions, webform/webform-id-2/view-submissions, ... and so on. But this produces so far lots of side effects in the routing/local task (aka tabs) subsystems which I am trying to solve now.

I've got the code pushed into a sandbox. https://www.drupal.org/sandbox/bucefal91/2838128n

It does the normal (what I find to be logical) integration between webform submissions and Views (fields, sorting, filtering). The only outstanding item that is worth mentioning is one problem: different webforms have different fields (webform elements). Then it's also logical to create a table of webform submissions as a tab on /webform/{webform}/submissions.

This leads to a problem: the same Symfony route must invoke different views depending on which webform it is, i.e. for webform1 we want to invoke a view where the fields are set up for webform1, whereas for webform2 we want to invoke another view where the fields are set up for webform2.

With some extra coding and investigation about how routing and local tasks are generated I was able to achieve decent result without making much of ugly deeds. My recipe is the following:

1. Create a view for each webform, set it up on /webform/webform1/submission path where webform1 must be actual webform ID and submission is any arbitrary chunk of path. Within the view set up a filter WHERE webform_id = 'webform1', i.e. hard code to only that specific webform. Set up the view to be a tab.
2. Repeat the procedure above for each webform.
3. webform_views module then will recognize the trick and will automatically integrate these paths with routing and local tasks.

For curious, to see how actually this routing & local tasks integration happens, feel free to check out \Drupal\webform_views\Routing\RouteSubscriber and \Drupal\webform_views\Plugin\Derivative\ViewsLocalTask classes.

I saw failed tests, but for some reason my local setup negated to execute webforms test cases (while executing fine all other tests) when running via simpletests and when trying to run via phpunit I discovered that webform tests use WebformTestBase which seems not to be supported by phpunit (they've introduced BrowserTestBase analog that is supported by phpunit).

Great! Feel free to see the source code of that sandbox and I'll be here to answer any of your questions.

It's great to see a views integration already started. I've pulled the sandbox repo down and am looking into it.

How can we get the ability to filter on select fields with the correct operator? It looks like we're running the filter on StringFilter. I actually don't know if we need another plugin/views/filter/filter.php file for the InOperator or if we can somehow change the WebformSubmissionFieldFilter.php to filter on several types of filters.

I think we might want to rename "Webform submission element value" back to "Webform submission data" to better match the D7 version, but maybe that's not important.

Also, do we want to back up and remove every field from showing up and show something like D7's "all values, value, value (raw)" instead? From there you can choose the field you want?

Since D8's webform allows you to select a webform to build a view off of, we should be able to pass that in and not have to worry about selecting the webform we want to work with, right?

As I tested a little bit more my code, I discovered a few additional bugs. Mostly it was about menu tab (aka local tasks). I've just pushed an update into my sandbox regarding the defects I found.

Also, right now the views integration does not cope well with webform pages. I'll see into it tomorrow.

owenmorrill, thanks for your feedback :) You see, I have nearly 0 experience with webforms, and misnaming the views labels to "Webform submission element value" instead of "Webform submission data" is a perfect example of how blind I am about the module's context :) I've fixed it.

About your question on filtering via "InOperator" - should be possible to achieve be it within the existing filter plugin or creating a new one. I'll see which way is the most convenient and will code it.

I do not think I've entirely understood your question about "Also, do we want to back up and remove every field from showing up and show something like D7's "all values, value, value (raw)" instead?" (probably again my little experience with webforms module limits my comprehension capacity). The way I take it is that you suggest to remove all the fields and instead just have 3 fields (all values, value, raw value) and within options (settings) of any of those 3 fields to specify which exact webform element we are talking about? I am slightly against this approach. I think Views' vision is to have 1 views field for each field whatsoever. This approach is executed by Field API - they create 1 views field for each field attached to an entity type. Apart from theoretical reasons, I also have one 1 practical: views code uses field label in the admin UI as the default admin title which can be manually overwritten to something else. Having distinct admin titles helps to grasp the overall picture about the view quicker in my personal opinion. The same title is also used as the default title for labeling the field in the UI (for "table" display it's the table column, for example). Again, if we keep those titles distinct it means a little less monkey job for people who create views (since they won't have to overwrite those default titles to make them more meaningful). I attach a screenshot where it's explained visually.

Since D8's webform allows you to select a webform to build a view off of, we should be able to pass that in and not have to worry about selecting the webform we want to work with, right?

Right now the solution I propose is to create N views for N webforms and within each view to specify a filter to include submissions only of that particular webform. When we get to the stage of wizard views creation, this step probably will be done automatically by the wizard, hehe, but just not yet :)

I guess really my question was centered on whether or not to show every field. For a smaller site that wouldn't be a problem--but it might get a little crazy with larger sites and however many hundreds of webform fields they'll create. Especially since, as far as i know, you can't re-use webform fields. At some point you'll need to be able to filter out everything but the fields on the webform you're working with. So you're looking at 5-9 fields instead of 100s.

I think I see what you're saying about including only the submissions of that particular webform--I assume you mean only the fields as well?

Is there a way to filter fields based on the add view "webform submission" of type "your webform"? (see attached screenshot) Because if we could use that to filter the fields--I think that could be handy. That way if you select of type "all" you would still get all of the fields but only the fields of "your webform" when you select that when creating a view.

I'm really new to Drupal 8, but I'm guessing there's a config entity value somewhere in views for "webform submission" for that "of type" selection. I'm also trying to look at the code for this, but learning how things changed from D7 to D8 is slowing me down--and I was already slow.

@bucefal91, before this goes too much further, you really need to take over https://www.drupal.org/project/webform_views. You won't be able to promote your sandbox into namespace used by an existing project. Because you already have a working version using that namespace and the existing project has been abandoned so long, it shouldn't take very long to get control of this if you follow the steps in https://www.drupal.org/node/251466

I see that you are already a co-maintainer of a few popular projects, but I couldn't find any full project nodes that you own. Only sandboxes. Have you gone through PAR?https://www.drupal.org/project/projectapplications

This might be a good opportunity to do that as well, but you should lay claim to the namespace first. PAR can take awhile, but you'll get the code in front of more people as well. After the reviews, you'll be able to create full projects yourself. You can submit the project after taking over https://www.drupal.org/project/webform_views. Doesn't have to be a sandbox to be reviewed.

Hello, @kreynen!

mmm, I was under impression my sandbox was a temporary home for the views integration code until it gets more or less reliable and then @jrockowitz would incorporate it into webforms module as a submodule.

But I do not have any preference, if the community prefers to have webforms_views as a separate project (as opposed to being a submodule of webforms project) that is fine by me. You tell me guys what you want :) My only bias in this question is the ease of maintainability: maybe webform_views will require to extend WebformElementInterface interface. And if it does require introducing new methods into that interface, then keeping it as a separate project may become a pain if feasible at all. Right now we are still doing just primitive things with views but as it evolves we actually might face the fact we need to ""amplify" some rather inner architecture of webforms.

I did pass the PAR. The "PAR can take awhile" is a very modest way to say how much while it can take. I do not recall it with precision but it took me like 6 or 9 months to get through it.

As far as code goes, I've just pushed a small update into the sandbox so that webforms pages are playing nice with the views too.

The part below I am addressing to comment #17.

At some point you'll need to be able to filter out everything but the fields on the webform you're working with. So you're looking at 5-9 fields instead of 100s.

Correct. Good thinking. How about instead of using very general and poorly descriptive "Webform submission data" we will do "Webform [webform-label] submission data". That way we will achieve what you're saying leveraging the pre-existing views UI (using pre-existing views UI should help with keeping low the learning curve). At the same time we must admit sometimes people will want to "span" their view over submissions from multiple webforms, so we also should support the scenario where fields from distinct webforms are available within the same view.

This change is really easy to code so I went ahead and actually updated the sandbox with this new notion of labels. If you have time and willingness, go try it. Probably you'll like it :)

Is there a way to filter fields based on the add view "webform submission" of type "your webform"?

I do understand your question and probably also comprehend the motives for asking it. However, that goes a little bit against how Views module look at it so I'd rather just implement the suggestion from above (introducing the groups "Webform [webform-label] submission data").

I think I keep saying it in nearly every comment I post here: I am have nearly no experience with webforms, so if you think my logic or thinking is wrong for whatever reason, you can always argue against my ideas and if you give me some real world scenarios/examples of why I am wrong, you'll save yourself some hours, because I will get convinced of my wrong position when I see some real use cases that go against it.

There is a risk that another developer will request the webform_views project/namespace while you are trying to develop a plan for webform_views. An active project in this namespace would be bad.

Ultimately it isn't going to matter if webform_views becomes part of the main Webform project sooner, later or never. Unless the name of the module is going to change, someone needs to take ownership of the abondend https://www.drupal.org/project/webform_views project.

My 2¢ is that @bucefal91 should request ownership of the webform_views project now and move the sandbox code into the 8.x branch of the webform_views project. While developers can be expected to git clone from a sandbox, a large part of the Drupal community cannot. Because of the amount of work @jrockowitz is already dealing with, it's reasonable to wait for a stable release of webform_views that has a relatively large number of site installs before even considering merging it into the main Webform project. If it makes sense to merge, then the https://www.drupal.org/project/webform_views project namespace is still protected.

We have a need for this functionality at the University of Colorado Boulder. We can commit resources to move this forward. It will be easier to do that in a full project than a sandbox.

Hello, guys!

Yes, I've also been thinking about the dilemma of either doing webform_views as a submodule within webforms project or as a separate project. I also incline towards a separate webform_views project now.

Basically webforms by itself is a tremendous amount of work and putting webform_views into the same pot would just do things worse in terms of implied workload. My concern about doing webform_views as a standalone project was that at some point we will have to alter WebformElementInterface interface and introduce some methods there that are required for Views integration. But I think that we can also introduce the concept of "views_handler" for each webform element. Then we'd have WebformElementViewsHandlerInterface kind of interface where all views-related methods will be kept and that way we can detach the 2 interfaces from one another.

We seem to have a consensus that webform_views should go into a separate project. So I think I'll be applying for webform_views project ownership one of these days. I am somewhat overburden with work & activities in this stage of my life, but soon I should clear up a little bit of time. Hehe, as a matter of fact I have other d.org modules sitting here waiting for my attention and here I am, assuming even more maintenance work :)

We're willing to run with webform_views for awhile, but like everyone else we have a limited amount of time and would only be focussed on this until it met our immediate needs. I don't really want to own yet another D.O. project that isn't part of our core service, but I'm happy to dedicate time from our team of developers to co-maintain the webform_views project for the next 2-3 months until @bucefal91 gets back to it, a new maintainer who can remain more focussed on webform emerges and/or it makes sense to merge a stable webform_views back into the main webform project.

What I can't do is approve deploying sandbox code to production when I know the sandbox has a conflicting contrib project namespace that is going to cause serious problems in the future.

I feel like we're playing hot potato here with functionality everyone acknowledges is needed, but no one has time to own. Perfection would be a stable version of this already existing in the main webform project, but I hope we can avoiding hinder progress towards that goal.

#2841459: Transfer ownership of webform_views - I have applied for ownership of webform_views project.

kreynen, yes. We are definitely playing hot potato here :) I will do my best to bring Webform Views to a stable release and make it feature-rich.

Anyway, since nobody else has taken much action with views integration, I think what I can do is better than nothing. Your support and/or feedback and telling what/why you need views will be helpful to grasp better understanding of why people need such code.

And shifting this code into webform_views project namespace is again no hurt since that project has been dead for over 2 years now so it's not like we are stealing someone's piece of cake. Yet, as you said, running off a full project module is much better than off a sandbox.

@dddave has fixed #2841459: Transfer ownership of webform_views so as soon as there is a 8.x branch, we'll start opening issues and submitting patches.

Gentlemen, I now I invite everybody to file tickets against https://www.drupal.org/project/webform_views The only thing is that for some reason the "open new issue" button does not appear to me on the UI when I open https://www.drupal.org/project/webform_views but hopefully it's just a question of some caching since I've just moved its status from "obsolete" to "active development".

I've just pushed my current sandbox into that project's repo.

Should we close this issue as duplicate or "wont fix"?

jrockowitz, yes, I recognize webforms itself may mutate far away from what it's now. I'll keep this in mind permanently whenever I brainstorm anything about its views integration and will try not to rely on "too internal" things of webforms so you have the freedom to rearrange the internal implementations. Basically, I'll try to use as public API of webforms as possible, thereby keeping the 2 code bases as decoupled as possible. This way each of us will have relative freedom to rebuild everything in his codebase without affecting the other. So at least think hard when you define public (developer's) API :)

I am gonna close this ticket as fixed. It's been a few days of inactivity here and people already started filing issues against webform_views project.

Thank you, everybody, for participation. I hope webform_views will be of use to you guys!

Please, instead of commenting in this issue or re-opening it, just find an appropriate issue (or file a new one) in https://www.drupal.org/project/webform_views