Problem/Motivation

We have a website with more than 50000 nodes. After installing the module, the cron runs take extremely long. I noticed that the google_analytics_counter_cron() function alone runs for like 10 minutes. That's because it builds every time a huge "count" queue with all the published nodes on the website. The more content we'll have, the longer it would run. Is that neccessary?

Proposed resolution

Wouldn't it be faster if this function only produced 1 "count" queue element at each cron run? This 1 queue element could do the following:

  • First, it queries all the published node IDs in ascending order.
  • It always checks if there's still time to process the next node by checking the "Queue Time (in seconds)" config setting.
  • If there's still time to do another one, it would run the \Drupal::service('google_analytics_counter.app_manager')->gacUpdateStorage($data['nid'], $data['bundle'], $data['vid']); just like now.
  • After it's done with that node, it would check if there's still time to do another one etc.
  • If there's no time to do another one, then it saves the last processed node ID to the state. And then ends the work in that queue element.
  • During next cron run, the new "count" queue element would continue the work from the following to the saved node ID.
  • If the queue element processed all node IDs then it would remove the saved node ID from state. And that way during the next cron run, it would start to process all nodes again.

What's your opinion about that? Could it work?

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

Comments

kaszarobert created an issue. See original summary.

  • kaszarobert authored c7d060ab on 4.0.x
    Issue #3099047 by kaszarobert: Make google_analytics_counter_cron()...
kaszarobert’s picture

Status: Active » Fixed

I decided to alter the current implementation similar to my proposal from years ago. As timeouts mentioned in referenced issue #3159018: When we have a huge number of nodes the cron job does not work (504 gateway timeout) still happening. But instead of using state, we only create 1 queue item at a time until there're no more nodes to process. And we're doing this NID descending order.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.