Hello,

Sorry, i created issue not at the right place, i knew it, but I'd like to ask people who uses the paragraph module. Did they experience out of memory issues?

I got an production VPS, 2-cores, 2Gb RAM, SSD. It's running under Ubuntu 16, php7.0-fpm, LEMP stack. (php: memory_limit = 384MB, mysql: max_allowed_packet = 512M)
My dev env is same stack, but with 4Gb RAM (php: memory_limit = 256MB, mysql: max_allowed_packet = 16M).

Both server has nothing specific, I mean standard my.cnf and php.ini except written above variables.

Everything runs good, exept a big node with bunch of paragraphs. The node has 80 paragraphs (40 with single long text field and another 40 with single image upload widget). When I'm trying to delete paragraphs, the Drupal allow me to remove only 5-6 items and after that I'm getting errors in my server log:

FastCGI sent in stderr: "PHP message: PHP Fatal error: Allowed memory size of 402653184 bytes exhausted (tried to allocate 100638473 bytes)

After I increased a php memory to 512M, it allows me to remove 7-8 items, but it crushes again with:

# mysql.log

[ERROR] /usr/sbin/mysqld: Out of memory (Needed 50581744 bytes)

and of course:

# nginx.log
Uncaught PHP Exception PDOException: "SQLSTATE[HY000]: General error" at /var/www/domains/example.com/public/core/lib/Drupal/Core/Database/Statement.php line 122" while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: example.com, request: "POST /node/346/edit?q=/node/346/edit&destination=/admin/content&ajax_form=1&_wrapper_format=drupal_ajax&_wrapper_format=drupal_ajax HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.0-fpm.sock:", host: "example.com", referrer: "http://example.com/node/346/edit?destination=/admin/content"

It happens only on production server, on my dev i can make any CRUD operation many times with no errors.

So, I'm going to increase memory on production server up to 4Gb (hope it helps). But is 80 paragraphs require fore than 512MB php memory OR something wrong with my prod environment? What you think?

Comments

xcono created an issue. See original summary.

xcono’s picture

xcono commented
Issue summary: View changes
xcono’s picture

xcono commented
Issue summary: View changes
xcono’s picture

xcono commented
Issue summary: View changes
xcono’s picture

xcono commented
Issue summary: View changes
miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented

Hm, entity operations are "heavy" but should not be that heavy and we don't do too much in paragraphs that really eats that much memory IMHO.

I'm confused why your database tries to allocate 50 MB of memory.
Also PHP tries to allocate another 100MB memory at the size of 400MB.

It seems you accidentally have huge amounts of data that gets serialised / saved.
Do you have some other type of modul active, triggering some accidental "almost" recursions?
You would need to do debugging. IMHO you could also simply look at the sql statements that are created either on Drupal side or on the mysqld log.

yobottehg’s picture

yobottehg commented
Assigned: Unassigned » yobottehg

i'll take a stab on this and will try to replicate with blackfire / xhprof

xcono’s picture

xcono commented

Miro, Yobottehg, thank you very mush for your attention.

I'm trying to replicate it on my dev now.

I tried to blackfire on prod, but see nothing serious. Now I'm trying to replicate on dev with xhprof, but see nothing bad - everything works fine.

Now I'm sure it's not a paragraph related, it's the question of POST request (and it's size) to rebuild form after adding another field. And may be it's related to my env.

List of modules:

 Administration    Admin Toolbar (admin_toolbar)                            8.x-1.15        
 Administration    Admin Toolbar Extra Tools (admin_toolbar_tools)          8.x-1.15        
 Chaos tool suite  Chaos tools (ctools)                                     8.x-3.0-alpha26 
 Core              Actions (action)                                         8.1.2           
 Core              Block (block)                                            8.1.2           
 Core              Breakpoint (breakpoint)                                  8.1.2           
 Core              CKEditor (ckeditor)                                      8.1.2           
 Core              Configuration Manager (config)                           8.1.2           
 Core              Custom Block (block_content)                             8.1.2           
 Core              Custom Menu Links (menu_link_content)                    8.1.2           
 Core              Database Logging (dblog)                                 8.1.2           
 Core              Field (field)                                            8.1.2           
 Core              Field UI (field_ui)                                      8.1.2           
 Core              Filter (filter)                                          8.1.2           
 Core              Internal Dynamic Page Cache (dynamic_page_cache)         8.1.2           
 Core              Internal Page Cache (page_cache)                         8.1.2           
 Core              Menu UI (menu_ui)                                        8.1.2           
 Core              Node (node)                                              8.1.2           
 Core              Path (path)                                              8.1.2           
 Core              System (system)                                          8.1.2           
 Core              Taxonomy (taxonomy)                                      8.1.2           
 Core              Text Editor (editor)                                     8.1.2           
 Core              Toolbar (toolbar)                                        8.1.2           
 Core              User (user)                                              8.1.2           
 Core              Views (views)                                            8.1.2           
 Core              Views UI (views_ui)                                      8.1.2           
 Field types       Entity Reference Revisions (entity_reference_revisions)  8.x-1.0-rc7     
 Field types       File (file)                                              8.1.2           
 Field types       Geolocation (geolocation)                                8.x-1.5         
 Field types       Image (image)                                            8.1.2           
 Field types       Link (link)                                              8.1.2           
 Field types       Options (options)                                        8.1.2           
 Field types       TableField (tablefield)                                  8.x-2.x-dev     
 Field types       Text (text)                                              8.1.2           
 Field types       YouTube Field (youtube)                                  8.x-1.0-beta3   
 Fields            Field Group (field_group)                                8.x-1.0-rc4     
 Multilingual      Interface Translation (locale)                           8.1.2           
 Multilingual      Language (language)                                      8.1.2           
 Other             Configuration Update Base (config_update)                8.x-1.1         
 Other             Configuration Update Reports (config_update_ui)          8.x-1.1         
 Other             Pathauto (pathauto)                                      8.x-1.0-alpha3  
 Other             Simple hierarchical select (shs)                         8.x-1.0-alpha1  
 Other             Token (token)                                            8.x-1.0-alpha2  
 Paragraphs        Paragraphs (paragraphs)                                  8.x-1.0-rc5     
 SEO               Metatag (metatag)                                        8.x-1.0-beta9   
 SEO               Simple XML Sitemap (simple_sitemap)                      8.x-2.5         
 Web services      RESTful Web Services (rest)                              8.1.2           
 Web services      Serialization (serialization)                            8.1.2
yobottehg’s picture

yobottehg commented
Assigned: yobottehg » Unassigned

I tried with blackfire on local with php7.

I could not replicate it. tried with 150+ paragraphs and nesting on one node. Memory usage does not go over 40mb ....

unassigning now, if you have a way to replicate i'm in again.

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented
Status: Active » Postponed (maintainer needs more info)

Sounds pretty healthy, thx.
In that case, we can not investigate it until further inputs.

xcono’s picture

xcono commented

Dear friends, thank you for you attention.

I think it's my nginx/php-fpm 7.0.4 related issue. Will try to rollback nginx and php configs to default.

I've got some blackfire report. Looks like every time I add/remove a drupal field with paragraph (may be not only paragraph) the php process memory increased by another 6 MB. (On my dev env memory stable and no exceed 40-44MB with 100 paragraphs).
So, paragraphs is awesome, and I think it's time to close issue.

I've created a new node and tried to add some fields.

1. Paragraph with image file added (18.1 MB): link
2. Image uploaded to the paragraph (24.5 MB): link
Let's add second paragraph
3. Paragraph with image file added (30.3 MB): link
4. Image uploaded to the paragraph (37.0 MB): link
Let's add second paragraph
5. Paragraph with image file added (43.1 MB): link
6. Image uploaded to the paragraph (50.2 MB): link
And third paragraph
7. Paragraph with image file added (57.0 MB): link
6. Image uploaded to the paragraph (64.8 MB): link

P.S. It's so bad to live with no brain.

berdir’s picture

berdir
Primary language German
Location Switzerland
commented

What I find strange is that a huge amount of that is deep down in PDO::execute() calls that are coming from the cache backend.

If you follow that up a bit, you can see that it's all coming from FormCache.

Which means that what's so huge in your case is the is the form_state cache. We've had problems with that before in core, this seems a bit different.

We need to figure out why that's so big and what is stored there and how we can optimize it.

You mention that it happens after multiple operations, so I suppose we somehow keep existing data in there and constantly add to it.

xcono’s picture

xcono commented

Berdir, yes I saw execute statement and unserialize and actually I begun to blame 7.0.4 version of php.
Tomorrow I'm going to investigate the cache and upgrade to newer php version.

But on dev it's still works like a charm (php 7.0.6, default nginx, default php-fpm conf). Hmm... and cache disabled on dev.

Thank you very much!

xcono’s picture

xcono commented

To @yobottehg, @miro_dietiker and @Berdir who gave me a most valuable clue.

It's a happy end of story.

I explored a strlen of $form_state after serialization in DatabaseStorageExpirable::setWithExpire. The size changed as predictable, but there wasn't big difference. When I checked $form after encoding and... it was increased every time form was processed. I wrote cache on disk and found in .json all my sitemap.xml generated by simple_sitemap module.

I don't know yet if it related only to my installation, but the problem solved.

supertony’s picture

supertony commented

I am having a similar issue as well. Here's what my error says:

"Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 63086870 bytes) in /Users/afuentes/Development/testinganewiqair-dev/docroot/core/lib/Drupal/Core/Render/Renderer.php on line 451".

I increased the memory limit from 256MB to 512MB and that seems to have fixed the issue for one of the few test pages I'm creating.

I'm using a few "layers" of paragraph types on some of the content, I'm thinking that's the problem.

xcono’s picture

xcono commented

@supertony Are you using some specific fields in your paragraphs?

miro_dietiker’s picture

miro_dietiker
Location Switzerland
commented
Status: Postponed (maintainer needs more info) » Fixed

Marking fixed for #14.
Hope you also reported the issue in simple_sitemap. :-)

xcono’s picture

xcono commented

@miro_dietiker, Yes, they already fixed it in -dev.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.