The PHP docblock example of the drupal_merge() function added by this patch explains the problem. Since module_invoke_all() and Drupal.settings are affected by this, I believe this is critical.

Comments

te-brian’s picture

This will be an invaluable function IMO.

I like effulgentsia's comment, I think it adequately explains the functionality, though my gut feeling says to put the example output of drupal_merge before array_merge_recursive. If people are just glancing at the docs, they should see the actual output before the example of the old method. That thought is without me looking for other examples of this type of comment, to see if we have a standard pattern.

I can't count how many random versions of this functionality I have floating around in various modules. Any module that uses an associative array to handle settings, especially when it comes to exportables, will use this immediately.

duellj’s picture

@te-brian:
It looks like it's standard pattern to put the wrong code first: e.g. http://api.drupal.org/api/group/php_wrappers/7

I think that the function name should be a little more verbose. drupal_merge is a little vague; what's being merged, arrays, objects, databases, etc? drupal_array_merge() would be better, maybe even drupal_array_merge_recursive(), since it indicates exactly what the function is doing (too long though?)

damien tournoud’s picture

Priority: Critical » Normal
Status: Needs review » Needs work

By definition not critical (we shipped countless versions of Drupal with this bug, and I'm pretty sure some of those sites are working correctly :p), but we really want this.

+      // Renumber numeric keys as array_merge() does.
+      if (is_numeric($key)) {
+        $result[] = $value;
+      }

The comment should probably read: // Passthru numeric keys.

This needs to be checked: do we really want to passthru *all numeric keys* or just those that are in a numeric sequence? Probably only the latter.

+      // Recurse only when both values are arrays.
+      elseif (array_key_exists($key, $result) && is_array($result[$key]) && is_array($value)) {
+        $result[$key] = drupal_merge($result[$key], $value);
+      }

Comment should read: // Recurse when both values are arrays.

array_key_exists() is known to be slow. This should be is_array($value) && is_set($result[$key]) && is_array($result[$key]).

+      // Otherwise, use the latter value.
+      else {
+        $result[$key] = $value;
+      }

Comment should read: // The latter value is used, overriding any previous value.

fago’s picture

Oh, a really good improvement!

>I think that the function name should be a little more verbose. drupal_merge is a little vague; what's being merged, arrays, objects, databases, etc? drupal_array_merge() would be better, maybe even drupal_array_merge_recursive(), since it indicates exactly what the function is doing (too long though?)

I second that. drupal_array_merge is more descriptive.

>The comment should probably read: // Passthru numeric keys.
But they aren't just passed through, they are re-numbered.

Also I wonder whether this might have a performance impact, as it's used in module_invoke_all()...

effulgentsia’s picture

Title: array_merge_recursive() is never what we want in Drupal: add a drupal_merge() function instead. » array_merge_recursive() is never what we want in Drupal: add a drupal_array_merge_recursive() function instead.
Status: Needs work » Needs review
StatusFileSize
new15.27 KB

Changed the function name to drupal_array_merge_recursive() since it's a replacement for array_merge_recursive(), similar to drupal_strlen() being a replacement for strlen().

Implemented feedback from #3, except for the first one, since as #4 points out, we *are* renumbering the keys.

This needs to be checked: do we really want to passthru *all numeric keys* or just those that are in a numeric sequence? Probably only the latter.

The implementation here is the same as what array_merge_recursive() does with respect to that. I'm not sure it's worth changing that aspect of the function (performance implications of checking whether they key is sequential?). update_retrieve_dependencies() is an example where the renumbering of all numeric keys is unfortunate, but even if we made the drupal_array_merge_recursive() function smarter, we wouldn't be able to use it here, since this function has additional logic controlling the merge (only use latter value if it is greater).

Also I wonder whether this might have a performance impact, as it's used in module_invoke_all()...

Yes, it does. For example, a user with permission to view comments and access user profiles viewing a node with 50 comments, results in drupal_array_merge_recursive() running 80 times (that's based on using devel_generate, which makes some comments authorship "root", having a link to a user profile and therefore a call to drupal_array_merge_recursive(), while some comments authorship "devel_generate" which does not result in a call to drupal_array_merge_recursive()). And drupal_array_merge_recursive() has a minimum overhead of 2us (on my machine) compared to array_merge_recursive() (more depending on the arguments), so that contributes 160us. Also, module_invoke_all('element_info') is 260us with the patch compared to 160us with HEAD. That's done once per page request, contributing another 100us. I haven't tracked down the other major sources, but the node with 50 comments scenario is 0.5ms slower (196.0ms HEAD, 196.5ms patch, +0.25%). IMO: this is worth it, since the behavior in HEAD is a bug, but I'm not sure what other profiling we want to do before deciding that.

effulgentsia’s picture

StatusFileSize
new15.38 KB
+++ includes/bootstrap.inc	7 May 2010 15:51:22 -0000
@@ -1783,6 +1783,52 @@ function drupal_hash_base64($data) {
+      // Renumber numeric keys as array_merge_recursive() does.
+      if (is_numeric($key)) {
+        $result[] = $value;
+      }

This patch replaces this with is_integer().

+++ includes/bootstrap.inc	7 May 2010 15:51:22 -0000
@@ -1783,6 +1783,52 @@ function drupal_hash_base64($data) {
+      // Recurse when both values are arrays.
+      elseif (is_array($value) && isset($result[$key]) && is_array($result[$key])) {
+        $result[$key] = drupal_array_merge_recursive($result[$key], $value);
+      }

This is an inefficient order. This patch moves is_array($value) to the end, resulting in shaving 20us from module_invoke_all('element_info').

84 critical left. Go review some!

damien tournoud’s picture

Note that #6 only introduces performance improvements. The semantics of the function remain the same.

damien tournoud’s picture

Alternative approach, that might be more efficient:

- first do an array_merge() of all the input arguments; you get an array where numeric keys are renumbered and containing only one entry for each string key (the value of the last argument containing that key)
- iterate over the keys of the resulting array, and for the string keys that correspond to an array value: collect the values of that string key in each of the arguments and call itself recursively on the result

This way we do most of the heavy iterating in C-code, and only iterate over what we really care about.

effulgentsia’s picture

StatusFileSize
new16.5 KB

The idea in #8 is interesting, but I wasn't able to come up with an implementation that's any faster than what's in this patch, at least for the module_invoke_all('element_info') case. I believe the reason is that most hooks like 'element_info' don't end up actually requiring recursion (multiple modules don't usually implement the same top-level key), but iteration through all the top-level keys and checking if the key is an integer or string is required in any solution, and that's the bulk of the time. Perhaps there are cases where #8 would be faster than this patch, but we'd need to determine what they are. But I think info hooks like module_invoke_all('element_info') are the ones that matter most, and this patch is the fastest for those.

This patch is a significant optimization relative to #6. It adds a drupal_array_merge_recursive_array() function to avoid having to call 'call_user_func_array' to construct Drupal.settings, and to avoid having to call func_get_args() during recursion. This also allows module_invoke_all() to call drupal_array_merge_recursive_array() just once, instead of drupal_array_merge_recursive() once for each module.

module_invoke_all('element_info') is 164us in HEAD, 184us with this patch.

effulgentsia’s picture

StatusFileSize
new16.57 KB
+++ includes/module.inc	11 May 2010 17:42:14 -0000
@@ -719,14 +720,12 @@ function module_invoke_all() {
+  $return = drupal_array_merge_recursive_array($return);

This patch adds an if(!empty($return)) check to avoid an extra stack call for hooks that don't return anything.

87 critical left. Go review some!

mtlhtml’s picture

Anyone can take a look at my issue ? Still... no response

Thx

effulgentsia’s picture

@mtlhtml: Was #11 meant for a different issue?

mtlhtml’s picture

Hi,

yes, different issue, i'm sorry about that.

I just dont know anymore where to post to get an answer...

Can you help me ?

effulgentsia’s picture

@mtlhtml: I'm guessing you're refering to #794388: Clicking the "count" button triggers a SQL query error. I think you posted that issue correctly: as an issue for the project you think is where the problem is (Gallery Assist), marked as a "support request" as the category. If rather than seeking support, you believe you found a bug, you can change the category to "bug report". The difference is that a "support request" is something where you're not sure if something about your installation or the way you're using the module is responsible for the problem, whereas a "bug report" is where you can provide the module maintainer with specific steps about how he or she can reproduce the problem from a clean Drupal install.

If you're new to working with open source software, one of the things you may find frustrating is that you don't get timely answers to your questions. This is often also true with proprietary software, but at least when you're buying software from a company, there's a tech support phone line or email address. But in the case of Drupal, module maintainers and people who participate in the support forums and issue queue often are not paid to do so, and need to carve out time in which to contribute as something they do in addition to their "day job". So, 2 days going by without an answer to a support request is not at all unusual. You just need to be patient. That said, check out the "Get Help" column of http://drupal.org/support for links to other places you can try. In addition to the issue queue of the module you're struggling with, you can try the support forums, IRC, email lists, and local user groups (all linked to from that page). You can also help attract faster support by offering a bounty.

In any case, posting to unrelated issues is generally frowned upon. You'll get better results from those other channels, so please stick to those. Thanks! A way to think of the Drupal community is that it's decentralized. People help out where they want to help out. People helping out on one project's issue queue may not be interested in helping out on another one. So posting on an unrelated issue is basically the same thing as SPAM. The best way to get help is to solicit it from where it's relevant. If the decentralized nature of open source is frustrating, you can always buy support from Acquia or another company.

mtlhtml’s picture

Hi, thanks a lot for taking time to answer.

Sorry for posting on an unrelated issue. Won't happen again.

I've been working with Drupal for a year now and i simply love it. I've struggled with a lot of issues that i took care by myself by reading all the posts in the drupal forum. This was the first time i post in one of these forums.

By the way, english is not my native language and there is a lot of words or phrase i still dont understand. Sometimes i have to search or read 5 times to understand and because of this, everything seems a little longer for me.

I'll be more patient.

Thx.

Crell’s picture

I can see the use case for this function, and can even think of places I may want to use it myself. However, I disagree with the name. This function has decidedly different semantics than array_merge_recursive(), which works perfectly fine for what it's documented to do. It should not be treated a broken function we have to wrap, which is what the drupal_* prefix implies: "Drupal's actually useful version of..." Maybe drupal_merge_array()? drupal_array_merge()? Something that doesn't imply array_merge_recursive is bad, since it's not. It's just different semantics than we want in some instances.

effulgentsia’s picture

Status: Needs review » Postponed

@Crell: I'm not sure the semantics of array_merge_recursive() is ever desirable, so I kind of do think of this as us needing to fix a broken PHP function, but I'll concede that that's opinion only, so your point is a good one. I renamed to drupal_array_merge_deep() and added it to the issue that originally inspired it: #823428: Impossible to alter node URLs efficiently.

So I'm postponing this issue until that one is resolved. Assuming that issue lands in some form, I'll re-open this issue to address the question of whether to use the new function for module_invoke_all() and Drupal.settings. I think we should, but as Damien points out in #3, it's not critical.

fago’s picture

chx’s picture

Isn't the result // This results in array('fragment' => 'y', 'attributes' => array('title' => t('Y'), 'class' => array('a', 'b', 'c', 'd'))). class is just array(c,d) ?

ksenzee’s picture

I posted drupal_array_merge_deep() at #208611-27: Add drupal_array_merge_deep() and drupal_array_merge_deep_array() to stop drupal_add_js() from adding settings twice as a solution to a pretty long-standing bug with drupal_add_js(). If and when it gets into core, we should revisit this issue. We had to write our own version of module_invoke_all() for Drupal Gardens to assemble a list of objects because array_merge_recursive() was corrupting the objects in the list.

@chx: No, items with numeric keys get added to the array; items with string keys replace existing entries. Just what we want.

rfay’s picture

subscribe

effulgentsia’s picture

More support related to #20: #208611-29: Add drupal_array_merge_deep() and drupal_array_merge_deep_array() to stop drupal_add_js() from adding settings twice. If anyone is able to, please help that issue land, so we can then discuss where else we need to switch from array_merge_recursive() to drupal_array_merge_deep(). Thanks!

fago’s picture

Version: 7.x-dev » 8.x-dev
Status: Postponed » Needs review
StatusFileSize
new533 bytes

We already have that function in d7 and d8 - so no cause for waiting anymore.

Status: Needs review » Needs work

The last submitted patch, drupal_array_merge.patch, failed testing.

tim.plunkett’s picture

Status: Needs work » Needs review
StatusFileSize
new553 bytes

Post /core reroll.

Status: Needs review » Needs work

The last submitted patch, drupal-791860-25.patch, failed testing.

tim.plunkett’s picture

The failing test was introduced by #721082: Prevent conflicting namespaces and move hook_rdf_namespaces() invocation into rdf.module, which is directly related to array_merge_recursive().

scor’s picture

We actually rely on the behavior of array_merge_recursive() for checking each RDF namespace associated with a given prefix in rdf_get_namespaces(), and ensure that they are all the same. The behavior of drupal_array_merge_deep() where the latter value replaces the former is not we want here, we want all the values, not just one. The patch #10 had the right hunk to fix rdf_get_namespaces() and was green.

tim.plunkett’s picture

Status: Needs work » Needs review
StatusFileSize
new2.78 KB

Rerolled #25 with the rdf hunks from #10.

scor’s picture

Status: Needs review » Needs work
+++ b/core/modules/rdf/rdf.module
@@ -672,7 +671,7 @@ function rdf_preprocess_username(&$variables) {
-  $variables['attributes_array'] = array_merge_recursive($variables['attributes_array'], $attributes);
+  $variables['attributes_array'] = drupal_array_merge_deep($variables['attributes_array'], $attributes);

This use of array_merge_recursive() should not be changed. (sorry, I missed it in #10). Afaict, this issue is about fixing the use array_merge_recursive() in places like module_invoke_all(), but not systematically across all core.

scor’s picture

Status: Needs work » Reviewed & tested by the community

after more conversation on IRC with @tim.plunkett, I'm retracting my previous comment, and RTBC'ing this.

tim.plunkett’s picture

catch’s picture

Status: Reviewed & tested by the community » Needs work

I'm not sure why we're doing module_invoke_all() and rdf module in the same issue, and then another follow-up to attack everything?

Also there's no tests here, but it's not impossible that someone tries to completely refactor module_invoke_all() and regresses it, so it'd be good to have a test for module_invoke_all() as well (or for this issue to explain why that doesn't need its own test coverage).

yched’s picture