I have field_collection field that contains hunderds of items. When adding them programmatically, first items are added fast but gradually it becomes slower. When there are already hundreds of items, adding one more field_collection_item takes SECONDS to complete. If I just use content types, it will be lightning faster.

Comments

AdamGerthel’s picture

I have the same problem. It gets gradually slower, in my case it gets slow after 3-5 field collections. After that it can take up to 10 seconds to add a new one.

kscheirer’s picture

Version: 7.x-1.0-beta4 » 7.x-1.x-dev
Priority: Normal » Minor
Status: Active » Postponed (maintainer needs more info)

Can you post the code you're using to programatically add items to the field_collection? Verification against the latest dev would help too, but I doubt that will make a difference.

AdamGerthel’s picture

@kscheirer my scenario might not fit the OP's. In my case it's simply when adding field collections in the regular manner on the node form. The field collection consist of a term reference field and a list (a couple of checkboxes).

chertzog’s picture

Status: Postponed (maintainer needs more info) » Active

EDIT - Turns out i had a had a bug in my code.

coreyp_1’s picture

Status: Postponed (maintainer needs more info) » Active

I have the same problem with slowdowns when creating nested field collections programmatically. I was able to solve some of my issues, but not all. My setup is something like the following:

container - container entity for everything else.
field_collection_1 - field collection containing 10+ fields as well as another level of field collection
field_collection_2 - second level of field collection, containing 10+ fields.

A rough layout of the end structure is:
container -> field_collection_1 -> field_collection_2

Simplified code:

$container = entity_create('entity_type', array('type' => 'entity_bundle')); // replace entity_type and entity_bundle with your info
// Set default values for fields in $container
$container->save(); // necessary in order to attach field collections

for ($i = 0; $i < $limit1; $i++) {
  $field_collection_1 = entity_create('field_collection_item', array('field_name' => 'field_collection_1'));
  // Set default values as necessary
  $field_collection_1->setHostEntity('entity_type', $container); // again, replace entity_type as appropriate
  $field_collection_1->save(TRUE); // calling this with TRUE is important!

  // Now, create multiple, second-level collections
  for ($j = 0; $j < $limit2; $j++) {
    $field_collection_2 = entity_create('field_collection_item', array('field_name' => 'field_collection_2'));
    // Set default values as necessary
    $field_collection_2->setHostEntity('field_collection_item', $field_collection_1);
    $field_collection_2->save(TRUE); // With the latest updates to Entity, this innermost save() may not be necessary.  Test on your own setup!
  }
}
$container->save(); // must be called again, otherwise the collections will not be added

The problem is, in just about every single example on the web, the field collection's save() is called directly, which defaults the parameter passed in to FALSE. This, in turn, causes the field collection's parent to be saved, which in turn re-saves *EVERYTHING* in the node. When creating multiple field collections (or, even worse, nested field collections as in this example) the results are disastrous with each new addition taking longer than its predecessor!

In my code, if I do not put TRUE into the save() calls, the script generates ~26,000 SQL queries. With the TRUE, it is down to around 2,300. Still not good, but a lot better than before. It appears that the majority of these are cache deletes and sets when the $container->save() is called the second time. I do not know a way around this. I wish I knew of a way to disable the cache for the few seconds that it takes to generate the node, but I haven't figured out a way yet.

I really wish we could improve the performance of this action, or perhaps figure out another way to accomplish the same task. Field Collection is a great module, but this difficulty limits the places in which it can be used.

EDIT: adding a conclusion because I don't think I was long-winded enough before. :)

I still think that this is a bug in that everything gets re-saved unnecessarily, but its impact can be lessened by appropriately using save(TRUE) instead of save() when programmatically creating multiple items.

kscheirer’s picture

Priority: Minor » Normal
Status: Active » Postponed (maintainer needs more info)

That seems like a decent workaround for now, thanks for the detailed writeup corey_p1! Setting to postponed for now - but what adjustments can we make to the field_collection module to improve this in the future?

coreyp_1’s picture

Oddly enough, it looks like updating Entity to the latest dev greatly reduced the number of sql queries, so we may be able to close this. (For future reference, Entity is currently on 7.x-1.0.) Updating Entity, in conjunction with appropriate use of save(TRUE) seems to fix the majority of performance and resource issues. This should probably be documented somewhere.

earwax’s picture

Status: Active » Postponed (maintainer needs more info)

I don't know if it is worth closing this issue yet. I updated to the the Entity API 7.x-1.1 and the performance is definitely better, but something slow still happens on the browser (front end).

I am using Firefox 20 and 21. When I hit the "add another item" button, then the "please wait" spinning logo spins and then stalls for like 3-5 seconds before the next fields pop up. If I use Internet Explorer 10, then as soon as the "please wait" spinning logo stops, I immediately get the next field(s) in my field collection. In other words, Internet Explorer 10 displays the fields in a field collection faster than Firefox version 20 and version 21. In Firefox, my entire browser just stalls. I have verified this behavior with another co-worker's computer also using Firefox.

I am just wondering if anyone else is experiencing this behavior and noticing the delay based on the different internet browsers. So for me, I don't think it's a database issue since I can patiently wait 1-3 seconds, but it's the additional waiting when the "please wait" spinning logo stalls the entire browser for another 3-5 seconds that irks me.

I am using Windows 7 as an end-user with Drupal 7.20 and field collection 7.x-1.0-beta5 on the backend (Apache server, Ubuntu 12.04)

AdamGerthel’s picture

@earwax the problem we've been having seems browser based as well. The computer's CPU goes up a lot when adding more, almost as if there's some javascript running that's causing loops or other unwanted behavior.

tkbcreative’s picture

@AdamGerthel @earwax - just wondering if you had any luck finding out what is going on here? I am getting the same problem with Firefox 21.0.

I have a field-collection including repeated-date fields, set to unlimited number of values, embedded in a node form which is using field groups.... and the "add another item" process is taking well over a minute, and maxing my computer's CPU. I'm running latest version of Entity, Field Groups and Field Collection. This obviously makes the UI essentially unusable in Firefox.

By way of a benchmark it is taking about 5-6 seconds to perform the same action in Chrome, which is fine from a usability perspective.

It does look like a Javascript loop/time-out issue, client-side, specifically in Firefox.... I wish I had the knowledge to help and debug.

Lukas von Blarer’s picture

Well, frontend performance is a different issue. File another issue report for that.

I had a lot of problems recently with performance for both creation and deletion of field collections. I also noticed a big issue with nested field collections regarding excessive revision creation and therefore unnecessary database growth which also affected the performance. What can be done to speed it up?

Which commit of entity api should improve performance? I didnt notice a big change.

loparr’s picture

Hi, this problem is known and it is related to lots of fields on node edit page. It seems that it is a issue in core and so far the only solution would be to use multi-step forms as mentioned here. http://drupal.stackexchange.com/questions/27971/horrible-frontend-perfor...

coreyp_1’s picture

Thank you for the reference. Just for clarification to anyone reading this later, the OP's issue is not with the browser because everything is being done programmatically.

The browser performance problem that was addressed in #8 - #12 is a completely different issue, as noted by Lukas in #11, and, just as he suggests, should be filed as a separate (core) issue.

jmuzz’s picture

Issue summary: View changes
Status: Postponed (maintainer needs more info) » Closed (works as designed)

Based on #7 and #13.

pbattino’s picture

Just for unlucky people like me that have to use corporate proxies, let me tell my story.

We experienced 2 problems:
1) what's described in this issue, BUT ONLY ON WINDOWS! (any browser). On Linux, the performances are acceptable even if, clearly, every row added makes the process a bit slower but it's not really noticeable.

2) an additional problem: the refresh of the page after adding a row does not work any more after a certain number of rows are added, especially if there's a lot of content in them. Basically the form is broken.

Luckly, it worked well on a development environment so we could compare and see what was different.

It turned out that in our production environment if the response gets too big, it is returned truncated, so Drupal is unable to read it and to refresh the page to show the new row.
What seems to break the response is Squid (http proxy) version 3.1. In fact, we have no Squid in dev and everything works fine. This is not surprising since the response is sent as "Transfer-Encoding: chunked" and the ckunking is part of HTTP 1.1 that Squid 3.1 does not fully support (in fact the "Transfer-Encoding: chunked" in the header disappears when going through Squid)

Why is this relevant? Because of problem 1) : even without breaking the response, it seems that Squid creates problem in transmitting the response, so Windows browsers are incredibly slow and they end up in "not responding mode" for a while even when the response is sent entirely, then eventually they resuscitate and the form is OK.

So perhaps some of you guys are in the same situation.

We are now testing with Squid 3.5, that claims full http 1.1 support, and PROBLEM 1) DISAPPEARED. The header now reports "Transfer-Encoding: chunked" correctly.

Unfortunately problem 2) persists, so there is still something wrong in how Squid interacts with the rest of the stack.