UPDATE (2015-10-29) ---- Original issue summary can be found in the revisions
------------
The cron lock time can be outrun by the max execution time.
A recent patch to the Drupal core enabled the possibility to have PHP scripts for a very long time, because the max_execution_time is not updated if it has been 0 before.
Still the hard coded value for the cron lock is 240 seconds. The problem with that is, that a cron, that runs over this amount of time is not cancelled, which is fine, of course, because some of them just take time.
That way a cron can be executed multiple times simultaneously which would trigger a lot of queue processing.
Comments
Comment #1
func0der commentedThis only fixes the issue with PHPCLI.
Not the fact that using a hardcoded value for such things is highly bad coding.
Comment #2
func0der commentedComment #3
Jooblay.net commentedCan you split with quary into subdomains, what about bash script or rsync then importing them into your drupal site.
Cron is more of a trigger then a lifter. If you could have your cron trip a script and then load all the nodes with feeds. Not sure...
Comment #4
func0der commentedFirst patch was broken. The lock time was not set if the time limit is between 0 and 240.
I also added a constant for the lock time.
Adding a constant or a configuration for the cron timeout is up to you, guys.
Comment #5
func0der commented@leseen:
I do not know, what you are talking about.
This:
was just for outlining.
Regardless of my case the fixed setting of a timeout is really bad practice and should also not make it into any final versions of open source cod. Especially not for a CMS with such spreaded needs as Drupal.
Regards
func0der
Comment #7
Jooblay.net commentedSorry for not being clearer. You want to collect 100k nodes from the greater web?
Have you viewed the feeds module and or sub-feeds modules. We import about 10k nodes on some commerce stores.
If you have the urls of the pages you are trying to quary using php and bash maybe the fastest way with little tools like aria2, rsync.
The you can build a bash script to load the urls, quary the data, and then load it via the feeds module with mysql database.
Sorry if none of this helps out...
Comment #8
func0der commentedI am sorry, that you missunderstood.
I am getting these nodes from MY database. No external feed or something. Just plain mysql server.
That is the problem.
But thank you for your interest and help.
Never the less, a timelimit should not be forced if it set to unlimited.
Comment #9
func0der commentedComment #10
func0der commentedComment #11
Jooblay.net commentedYou may find https://drupal.org/project/feeds_db of use?
Comment #12
Jooblay.net commentedAlso just a note I dont think this ticket meets the critical queue?
Comment #13
func0der commentedNo, but thanks.
Also that module is in alpha and for 7.1.
Comment #14
mgiffordI don't know if this is critical, but you've got some good questions. Your main concern is about the hard coded timeout in includes/common.inc drupal_set_time_limit(240);
This also occurs in Drupal 8 and would need to get fixed there and backported to D7:
core/lib/Drupal/Core/Cron.php: drupal_set_time_limit(240);
core/modules/node/node.module: drupal_set_time_limit(240);
Comment #15
jhedstromCron execution was moved to a service, so this patch would need to be against that (#2169267: Replace drupal_cron_run() with a Cron service).
For the larger issue though, Drupal 7 introduced a job queue, and adding really large tasks to a custom queue, and then processing directly with something like drush will probably work much better than the automated cron system.
Comment #16
swentel commentedSee #547998: Make cron time limit a variable
Comment #17
func0der commented@swental: Yes, see https://www.drupal.org/node/547998.
That is not what THIS ticket is about.
This ticket is about NOT setting a time limit AT ALL, if we are on PHPCLI (@mgifford: this is my main concern. The hard coded part is a pain in the ass, too, but mainly this.).
The ticket you mentioned is about a configurable time limit for the cron hook. Two totally different things.
So. Re-opend.
Comment #18
swentel commentedWell, they are definitely related because that one will allow you to set it to anything and configurable through the UI, and overrible in code easily and (imo) cleaner code. Anyway, won't fight over this.
Also, for long operations like this, I usually go for drush scripts and batches - unless you're really stuck to hook_cron of course.
Comment #19
func0der commentedOkay, let's get away from the example I gave to just the fact that making something from good to worse is bad.
In clear: setting the max execution time to something if it is indefinite is just not good practice.
Setting it over the UI is not helping here in any way. Because setting it to 0 in the UI would either result in an error during execution or worse in very, very, very long waiting times for users, it is run as a web hook and not via PHPCLI/drush.
Comment #20
func0der commentedFixed with this #2233929
Comment #21
func0der commentedComment #22
func0der commentedNevermind. This is not fixed, because the cron is still stuck with the time limit of 240.
It gets in fact even worse with the patch in #2233929, because now crons will run simultaneously.
I am going to fix this in a new patch.
Comment #23
cilefen commented@func0der
I am tagging this Needs title update and Needs issue summary update because the title and issue summaries need updating to reflect the fundamental problem—something like: "max execution time in hook_cron is hardcoded".
Comment #24
func0der commentedComment #25
func0der commentedI have just realized that my patch probably does not fix the issue of multiple queues starting at the same time, because that part of the source code is excluded from the cron lock.
I guess we have to think of something else here.
@cilefen: Thanks for making me rewrite the issue summary ;)
Comment #26
poker10 commentedThanks for working on this. I think this was fixed by #3308929: [D7] Cron lock time limit is too short and does not prevent multiple, concurrent cron runs. Cron lock is no longer a fixed value, instead its value is from a variable. Feel free to reopen if this is still an issue.