Memory leaks in getVariations method [#2938577]

I created a lot of variations (over 1000) for one product. When I call $product->getVariations() method, then memory usage grows very fast and memory does not become free. Never.
Simple example (executes in PHP-devel window):

$mem = false; // Show allocated memory
dvm(memory_get_usage($mem), 'Start');
$product = \Drupal::entityTypeManager()->getStorage('commerce_product')->load(5); // Load a product
dvm(memory_get_usage($mem), 'Product loaded');
$variations = $product->getVariations(); // Load variations
dvm(memory_get_usage($mem), 'Variations loaded');
dvm(count($variations), 'Variations count'); // Count of loaded variations
dvm(memory_get_usage($mem), 'Before unset');
unset($product); // Unset product variable
dvm(memory_get_usage($mem), 'Product unsetted');
// debug_zval_dump($variations);
unset($variations); // Unset variations variable
dvm(memory_get_usage($mem), 'Variations unsetted');
$gc = gc_collect_cycles();
dvm(memory_get_usage($mem). ' GC = '. $gc, 'Finish'); // WTF?!

Result of execution:

Start => 7108168
Product loaded => 8272648
Variations loaded => 49625800
Variations count => 1612
Before unset => 49626056
Product unsetted => 49626176
Variations unsetted => 49552512
Finish => '49552648 GC = 0'

Drupal 8.4.4, PHP 7, Zend Memory Manager = enabled, zend.enable_gc = On
I suppose it is a global problem of Drupal due to call trace of class methods ups to core methods such as

class CommerceContentEntityBase extends ContentEntityBase implements CommerceContentEntityInterface {
  public function getTranslatedReferencedEntities($field_name) {
    $referenced_entities = $this->get($field_name)->referencedEntities();
    return $this->ensureTranslations($referenced_entities);
  }

This problem also occurs when I save product after deleting or creating variations. Memory peak is over 350 MB and it only grows. Never reduces... Unset() etc. no helps.
I have product with over 10000 variations and 512 MB memory for PHP is not enough... Even for deleting only one variation!!! Product->postSave goes over all variations and PHP falls.

Comments

Comment #1

22 January 2018 at 14:30

Yaroslav F.D. created an issue. See original summary.

Comment #2

Yaroslav F.D. CreditAttribution: Yaroslav F.D. commented 22 January 2018 at 14:33

Issue summary:

View changes

Comment #3

bojanz CreditAttribution: bojanz at Centarro commented 22 January 2018 at 14:41

I don't think we'll ever be able to support a product with 10000 variations (we never did in the D7 version either).
Most people have up to a 100, that's our default use case. More than that, and you'll want to rethink your product architecture (and move certain attributes to the order item).

That said, any performance bug is worth chasing. This one might need to be chased further up the stack, as you've noticed. If you open any core issues, link them here.

Comment #4

mglaman

English

WI, USA

CreditAttribution: mglaman as a volunteer commented 22 January 2018 at 16:00

I'd love to see some XHProf, Blackfire, or Tideway callstacks on this. I'm sure there's plenty of core improvements which can be done, but have yet to be flushed out.

Comment #5

Yaroslav F.D. CreditAttribution: Yaroslav F.D. commented 26 January 2018 at 16:48

Hi,
I tried to get some info about variables, memory, garbage from xdebug and found zero.
So I changed ProductVariationAttributesWidget.php to using SQL query for getting count of variations if it is called from view. Because view of product (Product Catalogue) eats memory and time as a moon beast.
Who thinks that my life is not enough boring, I may to expand my example.
I build webshop for selling tickets for entertainment. For 60 days from now, 26 sessions per day, 10 tickets per session. 15600 tickets for one type of entertainment. One ticket = one variation. This concept allows me to use base product "Add to cart" form with cascading product attributes: date, session, place. Also I always know is a variation sold, reserved or free, due to 3 additional fields for variation type.
This example is very heavy for current Drupal OOP core. 512 MB for PHP is sufficient only for 9000 variations. I dropped place attribute and use count of sold tickets field in each variations (<=10).
So I have only 1560 variations per product. Product view form with Add to cart button refreshes during 2-3 seconds. It is comfortable now.
But view with 10 products in list was built for 20-30 seconds. Therefore I changed file in Commerce module. If product has one variation then "Add to cart" button in product catalogue adds it to cart, else product page is opened and all product attributes are displayed.

I suppose that memory leak is a problem of Drupal core due to unnecessary enthusiasm to OOP using.

Comment #6

drugan CreditAttribution: drugan as a volunteer commented 26 January 2018 at 21:32

Did you test your products without "3 additional fields"? Also, did you consider to find more powerful server? I have apache (1024 MB) on my pretty old laptop and it quite decently woks with 1000 variation products.

Is it possible in your workflow to create variations by demand? For example, you create products each having let's say 500 variations and then create event subscriber and automatically add more variations to a product when they are almost sold?

Comment #7

Yaroslav F.D. CreditAttribution: Yaroslav F.D. commented 30 January 2018 at 12:28

Hi, Drugan!
I added 3 additional fields: store count, reserved count, sold count. Usually I set Store = 1 (when variation is created). When somebody adds variations to cart, then Reserved = 1. If somebody will not pay, then Reserved = 0 (after X minutes) and order item will be cleared. After payment Reserved = 0, Sold = 1. Simple schema.
Here is a screenshot of worked site on provider virtual server: http://pixs.ru/showimage/awproduct4_2712569_29188017.png
It is a variant with 260 variations per day. Too much.
Another mean is http://pixs.ru/showimage/awproduct8_2408989_29188073.png (26 variations per day).
I set Store = 10 and sell tickets until (Store - Reserved - Sold) > 0.
Store module is not ready for Drupal 8. I know :-) So I wrote small myself.

Comment #8

drugan CreditAttribution: drugan as a volunteer commented 30 January 2018 at 16:33

Now I see.. You support kinda your own stock algorithm. I wonder why you don't use the https://www.drupal.org/project/commerce_stock module? For now they have just dev D8 version but it works perfectly (at least for me). With the module you can write variation availability resolver and that's it. No any additional fields are needed. The only thing that you should fix is to check at the moment of adding variation to cart whether it is in stock right now. Of course, you should do the same when displaying variations (seats) because reserved (not purchased yet) should be marked in red, do they? But remember if you don't use ajax for this purpose (overkill with such a huge number of variations) then there is a lag between a customer downloads the page and actually presses Add to cart button. Theoretically, it might be quite long, even a couple of hours.

No problems! Enable Commerce Extended Quantity module, set up quantity field settings on an order item type and you'd have your availability resolver automatically called both on the Add to cart form and /cart page when updating the quantity. See more:

http://cgit.drupalcode.org/commerce_xquantity/tree/src/Form/XquantityAdd...
http://cgit.drupalcode.org/commerce_xquantity/tree/src/Plugin/views/fiel...

BTW, you can upload demo screenshots directly on this page by pressing Browse... button below. No any external services are needed.

*а то девочки на http://pixs.ru только дразнят гуся, отвлекая от полезного, производительного труда :). Да и страница у них очень долго грузится - не каждый выдержит столько ждать.

Comment #9

Yaroslav F.D. CreditAttribution: Yaroslav F.D. commented 1 February 2018 at 19:23

Thank you, Drugan! I knew about Commerce Stock, but I thought it is not ready after I had read issues of dev version. Now I am developing module to prevent criminal reservation of tickets, so I will redesign my webshop later.

* Шутку понял :) У меня адблокплюс стоит, я годами рекламу и девочек не вижу, поэтому не знал, что на пиксе творится. Я в первый раз им вообще пользовался, гугл подсунул.