Jobs kept behind schedule

One reason could be that you have cronjobs taking a long time to finish, which makes the cronjob in question miss its schedule due to the "catch up" setting.

How often is cron running (and is it poormans, crontab wget, etc)?

Can you provide me with the configuration of the job in question? If you have access to drush you can do a (example for the node_cron job):

drush cjget node_cron --fallback

If possible could you also provide me with:

drush cron-list

or, alternatively, provide me with a screendump of the jobs page?

Ok. Sounds much like the job is missing its launch window due to other jobs taking a long time to finish. The crontab experiments you've conducted would produce that result.

It takes only one job (job #1) to block another (job #2), if job #1 runs for over 5 minutes. This is a somewhat natural effect of Ultimate Crons throttling.

I have 2 potential solutions:

1.) Increase the launch window for the problematic job:
- Change scheduler to Crontab for the job
- Set the rule to 0+@ * * * * * (every hour)
- Set the catch up to 3600 seconds (one hour)

This will ensure that the job has a chance to catch up.

2.) Increase the throughput of Ultimate Crons job processing.
- Ensure that cron is invoked every minute and that you're not using poormans cron (i.e. * * * * * in servers crontab).
- Increase number of threads to e.g. 2 or 3.

This will make Ultimate Cron able to process up to 2-3 jobs simultaneously if a job takes more than one minute to complete. The original problem can still occur, but can be greatly mitigated. This can be even more fine-tuned through selecting which thread each job should run in. For complete control of the parallel threads, setup an equivalent number of wgets in your crontab (see https://drupal.org/node/2195421).

Set the rule to 0+@ * * * * (every hour), 4 stars after @, not 5 - right?

Yes. 4 stars.

The schedule still remains quite uneven although more stable, like with the following pattern for one of those two jobs (it shows the skew of +45 picked - in the tooltip for the job).

This means that the jobs will attempt to be launched at 19:45, 20:45, 21:45, 22:45, etc. However, if you have another job that is also launched at, say 19:45, and that job is launched before the one you listed AND that job takes 21 minutes to finish, then this job we want launched at 0+45 will not launch until 20:06, because there's no slot available for it. This can be mitigated by my previous #2 suggestion, where the number of threads is increased.

All this is because the jobs are running consequetively (serial launcher).

Each minute Ultimate Cron determines which jobs should be run. The two jobs you are trying to fix are not probably the cause of the uneven schedule. The other jobs launched at the same minute are.

Tried it for two pretty light jobs (and the only two left enabled) in the testing configuration of Drupal not under active load.

The load of users on the site is irrelevant to the cron processing (unless the webserver has no sockets left for connections).

Would a better scheduling percision be possible, to have more equal intervals between launches of the job?
As far as I understand there is no common cron in Ultimate Cron, e.g. to set up a 'cron clock' checking schedules for all jobs wih e.g. 5-minute interval and launching those that are due at those 5-minute points? So in Ultimate Cron each job has its own clock, right? (trying to understand how the clock could maintain more equal launching intervals)

I'm not entirely sure what you mean here. Instead of the "+@", you could try to specify the minutes manually in order to tweak it. But the server capacity is will still be the final piece of the puzzle (suggestion #2 increasing threads).

To achieve full "precision" for all jobs, you must use Background Process launcher, set the threads for that launcher to (at least) the number of jobs present. This will most likely require a larger/more server(s).

As a comparison for core I can say; if you set all jobs to run at 0 */3 * * * with a very high catch up (e.g. 100 years), then Ultimate Cron will process the jobs in the same way that core would (with a 3 hour interval). You will just have a better overview of exactly when each job was run, how long it took, and error logs for each job.

The Background Process launcher will be available when installing the Background Process module. The number of threads in the serial launcher settings is only relevant for the serial launcher.

I recommend that you read through the documentation first (https://drupal.org/node/2195383). It's not that comprehensive, but it should provide a rough overview of the various parts and mechanisms in Ultimate Cron.

Thank you gielfeldt for your explanations that helped me solve a similar issue.