For reasons we have not yet been able to determine, we get jobs inserted to the database without a payload, randomly. These get stuck in processing and then error out. It gets constructed correctly, otherwise the Job constructor would error.
But yet we have a bad record saved
MariaDB [main]> select * from advancedqueue where payload = '';
+---------+------------+-----------------------------+---------+------------+---------+-------------+------------+-----------+------------+
| job_id | queue_id | type | payload | state | message | num_retries | available | processed | expires |
+---------+------------+-----------------------------+---------+------------+---------+-------------+------------+-----------+------------+
| 1758950 | order_jobs | JOB_TYPE | | processing | NULL | 0 | 1588427412 | 0 | 1588953615 |
+---------+------------+-----------------------------+---------+------------+---------+-------------+------------+-----------+------------+
Which gives us
/app/workers/../bin/drush advancedqueue:queue:process order_jobs: In Job.php line 99:
Missing property "payload"
🤷♂️ So it's fine when we do
$job = Job::create('JOB_TYPE', [
'order_id' => $order->id(),
'order_line_items' => $order_line_items,
]);
$queue->enqueueJob($job);
But for some reason it enqueued and writes incorrectly. The only thing I can think of is that JSON encoding somehow fails?
From the Database enqueueJobs:
$fields = $job->toArray();
unset($fields['id']);
$fields['payload'] = json_encode($fields['payload']);
// InsertQuery supports inserting multiple rows at once, which is faster,
// but that doesn't give us the inserted job IDs.
$query = $this->connection->insert('advancedqueue')->fields($fields);
Regardless, the following should cause the job to fail, not halt queue processing
public function __construct(array $definition) {
foreach (['type', 'payload', 'state'] as $required_property) {
if (empty($definition[$required_property])) {
throw new \InvalidArgumentException(sprintf('Missing property "%s"', $required_property));
}
}
Comments
Comment #2
mglamanI don't think there is a generic way to resolve in the Processor.
There's no way to handle the broken job and flag it. but if it fails to load, the queue just crashes.
In the Database queue backend, we should use a try/catch when creating a job object from the database values. If there is an exception, we can update that job as a failure and let the next job become picked.
Comment #3
lisastreeter commentedI've confirmed that malformed UTF-8 characters in the payload causes empty payloads.
json_encodereturns FALSE when an array can't be encoded. I've added a check in the backendenqueueJobs()that will set the Job state to FAILED (with a message about the malformed payload) instead of QUEUED if the JSON encoded payload is empty.I also created a SupportsPayloadValidationInterface interface with a
validatePayload()method. Services that create Jobs can use the method to validate the payload before adding the job the queue, to avoid the FAILED jobs and employ some other sort of error handling.Comment #4
mglamanWe could use json_last_error (https://www.php.net/manual/en/function.json-last-error.php) and json_last_error_msg (https://www.php.net/manual/en/function.json-last-error-msg.php)
If the JSON payload fails, we could throw an exception / error when enqueuing the job.
So
$queue->enqueueJob($job);would throw an exception or mark the job as immediately failed if JSON encoding encountered an error.Comment #5
mglamanAn example is from \GuzzleHttp\json_encode.
Comment #6
bojanz commentedI like mglaman's idea.
Comment #9
c_archer commentedRan into this issue, have created a patch implementing mglaman's idea