Problem/Motivation
Currently taking 15 seconds to load a page on new production server. We went from a single tier to a two tier architecture.
In our dev environment we have a single tier application with the web/dev server together. Page loads were less than 5 seconds and OK. We recently moved to our production environment that has separated the two servers and performance has taken a significant hit. I know that there are hundreds of reasons why performance could be bad but I wanted to put up this information just in case there are any lessons to learn.
All the images are from newRelic a performance tracking tool (scroll to the last image to see yamlforms 5 second load):
Here is a table of load times for the three times renderPlaceholderFormAction (rPFA) ran.

Here is a breakdown of time by each component.

A high level stack trace that shows how much time each call tacks.

A more granular look at yamlforms time contribution.

Proposed resolution
Remaining tasks
User interface changes
API changes
Data model changes
| Comment | File | Size | Author |
|---|---|---|---|
| #21 | Screen Shot 2016-11-02 at 9.34.41 AM.png | 1.35 MB | robpowell |
| #15 | Screen Shot 2016-11-02 at 1.06.06 AM.png | 214.18 KB | robpowell |
| #15 | Screen Shot 2016-11-02 at 1.05.53 AM.png | 132.67 KB | robpowell |
| #15 | Screen Shot 2016-11-02 at 1.03.37 AM.png | 146.5 KB | robpowell |
| #15 | Screen Shot 2016-11-02 at 1.03.21 AM.png | 90.27 KB | robpowell |
Comments
Comment #2
jrockowitz commentedI am seeing amajor issue where the YamlForm entity is not being cached because it is a config entity and this is causing its elements are being initialized multiple times. This would lead to a performance issue for very large forms.
This is going to take a little work to figure out the best solution.
Comment #3
jrockowitz commentedComment #4
jrockowitz commentedYep, this is an issue that is being worked on in Core.
#1885830: Enable static caching for config entities
My best option is probably going to be implement a temporary static cache for just a YAML Form's elements.
@robpowell There is no guarantee that this will fix your performance issue. I will look into starting to benchmark the YAML Form module.
Comment #5
robpowellThank you, I am glad you found the screenshot helpful. Please let me know if there is any more information I can provide to push this along. This form is a wizard page form with approximately 75 fields.
Comment #7
jrockowitz commentedLet's see if the attached patch passes all the tests.
I don't have an immediate benchmarks but the patch does prevent multiple YAML Form config entities loads when submitting a form.
Comment #10
jrockowitz commentedComment #12
jrockowitz commentedI added a new test that confirms single YamlForm::load() calls are cached while YamlForm::loadMultiple() calls are not cached.
Comment #13
jrockowitz commented@robpowell can you please test the patch? BTW, is Bigpipe enabled on your site.
If you don't have time to test the patch, I can just commit this patch since the tweaks are very basic and include test coverage.
Comment #14
robpowellI can test in about three hours, at Boston meet up.
Comment #15
robpowellSo I applied the patch and via the new relic metric it looks like we shaved three seconds. Attached are the images, please let me know if you need different info. Also, do you think memcache would help? We do not have big pipeline enabled.
Comment #16
jrockowitz commentedMemcache (or Redis) is a must if you want good performance. Varnish is also a good idea but it won't help much with a YAML Form's performance. Using PHP7 with the YAML extension will also boost performance.
The below URL might help you a little...
http://drupal.stackexchange.com/questions/217274/why-drupal-core-form-fo...
How are other pages performing in your 2 tier architecture? How is the YAML Form module's default Contact form performing?
For local development, I am going to try to install Blackfire.io and see what metrics I can capture.
I am going to commit the patch but keep this ticket open.
Comment #17
jrockowitz commentedChanging title to be more generic and include the word performance.
Comment #19
jrockowitz commented@robpowell I created #2824272: Add wizard test forms with 100, 200, and 300 element.
I definitely need you to benchmark other forms, especially the long test forms included in the yamlform_test.module.
You will need to add the below setting to your settings.php to be able to enable the yamlform_test module
Comment #20
robpowellGreat, let me review what I think are the next steps:
How would you like me to capture benchmarks? Are those screenshots helpful to you or do you need me to capture specific page load times? Anything you can suggest for me to get the results to you faster would be great.
Thanks!
Comment #21
robpowellSo looking over Memcache it looks like the big gains are for authenticated users, all of my users are anonymous so while there won't be immediate gains we will want it for the long term.
source https://www.youtube.com/watch?v=_gv7iZQowag
Varnish seems better suited for my use case because it handles the static cached pages for anonymous users. Just a thought, and keep in mind I am a drupal noobie, but is there any way to cache the form in Internal Page Cache? This seems promising ....
source https://www.drupal.org/docs/8/administering-drupal-8-site/internal-page-...
Regarding YAML extension, it looks like this is now part of core so what do I have to do to set it up? Make sure the php module is installed and update a settings file flag?
Comment #22
jrockowitz commentedMemcache always helps, even with anonymous user traffic. For example, any POST request will bypass Drupal internal page cache and Varnish's cache. Also Memcache will reduce PHP's memory usage which is a huge help for Apache (but not so much for NGINX).
For 2-tier custom server environment you are going to want to mirror Pantheon's and Acquia's caching stack. https://pantheon.io/features/advanced-caching.
Installing the PHP's YAML extension will result in your form's YAML being parsed much faster.
Also checkout this older video. https://drupalize.me/videos/overview-performance-scalability
Comment #23
robpowellthanks. working on getting that set up now.
Speaking about posts, can you talk me through the wizard_page post? Is the wizard_page doing a fully qualified request to the site? Meaning, if it is hosted on
http://foo-bar.org/does the form request the url http://foo-bar.org/form/foo after each click of the next button?The reason I ask, some more information came to light about the network architecture and it turns out that we are behind a F5 load balancer and any fully qualified url goes back to the device before coming back to the page. At the very least, it is a concern their dev ops team has and I would like to ease their concerns and put it to bed.
I'll have some updated tests shortly.
Comment #24
jrockowitz commentedIn Drupal, anytime a form is submitted (aka a POST request) the page cache is bypassed and the form's submit handler is processed. The YAML Form wizard does trigger a rebuild of the entire form because the submitted and displayed has changed.
It is worth noting that when page's cache is bypassed and in our case a form is rebuilt, Drupal is still using caching for things like entities and especially blocks and this is where you get decent performance improvement by using Memcache since the cached blocks are pulled from Memcache (memory) and not the database (disk).
Right now you should just benchmark page load times and memory usage. I don't want to look at the load stack until I feel there is any issue.
A few pages worth benchmarking as anonymous (and authenticated when possible) ...
It is also worth comparing the Bartik themes performance to your custom themes performance, just incase there is a performance issue at your theme layer.
Comment #25
jrockowitz commented@robpowell Did you workout your performance issues?
Comment #26
robpowellThanks for the followup. It turns out the new hosted environment had load balancing. Apparently the way this LB was set up was the two apps servers had one "shared memory' directory. Once we moved to having the files all on the app server we saw significant reduction in page load times. Currently, we are seeing about 2-5 seconds per page and memcache isn't set up yet.
Sorry it took so long to get back.
Comment #27
jrockowitz commentedGlad to hear that you figured out the issue.