Problem/Motivation

Every once in a while we get Operation timed out exceptions when trying to connect to the ccu endpoint. These end up being unhandled exceptions and bubble up the stack. For normal editor traffic these end up in the php error log. (This probably also means that those items are lost from queue)

This really causes issues for us when deploying our site as one of our Drush commands in our deployment script sees this as an issue and returns a non-zero status code. (It looks like it depends on when the error is captured as to which command will have issues) 

GuzzleHttp\Exception\ConnectException: [17-Nov-2025 09:37:26 America/New_York] GuzzleHttp\Exception\ConnectException: cURL error 28: Operation timed out after 5001 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://xxxx.luna.akamaiapis.net/ccu/v3/delete/tag/production in /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php on line 275 
#0 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(205): GuzzleHttp\Handler\CurlFactory::createRejection(Object(GuzzleHttp\Handler\EasyHandle), Array)
#1 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(157): GuzzleHttp\Handler\CurlFactory::finishError(Object(GuzzleHttp\Handler\CurlHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
#2 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(47): GuzzleHttp\Handler\CurlFactory::finish(Object(GuzzleHttp\Handler\CurlHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
#3 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(28): GuzzleHttp\Handler\CurlHandler->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#4 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Handler/Proxy.php(48): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#5 /mnt/www/html/ncigov01live/vendor/akamai-open/edgegrid-client/src/Handler/Authentication.php(83): GuzzleHttp\Handler\Proxy::GuzzleHttp\Handler\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#6 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php(64): Akamai\Open\EdgeGrid\Handler\Authentication->Akamai\Open\EdgeGrid\Handler\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#7 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Middleware.php(31): GuzzleHttp\PrepareBodyMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#8 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/RedirectMiddleware.php(71): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#9 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Middleware.php(66): GuzzleHttp\RedirectMiddleware->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#10 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/HandlerStack.php(75): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Request), Array)
#11 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Client.php(333): GuzzleHttp\HandlerStack->__invoke(Object(GuzzleHttp\Psr7\Request), Array)
#12 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Client.php(169): GuzzleHttp\Client->transfer(Object(GuzzleHttp\Psr7\Request), Array)
#13 /mnt/www/html/ncigov01live/vendor/guzzlehttp/guzzle/src/Client.php(189): GuzzleHttp\Client->requestAsync('POST', Object(GuzzleHttp\Psr7\Uri), Array)
#14 /mnt/www/html/ncigov01live/vendor/akamai-open/edgegrid-client/src/Client.php(152): GuzzleHttp\Client->request('POST', '/ccu/v3/delete/...', Array)
#15 /mnt/www/html/ncigov01live/docroot/modules/contrib/akamai/src/Plugin/Client/AkamaiClientV3.php(61): Akamai\Open\EdgeGrid\Client->request('POST', '/ccu/v3/delete/...', Array)
#16 /mnt/www/html/ncigov01live/docroot/modules/contrib/akamai/src/AkamaiClientBase.php(288): Drupal\akamai\Plugin\Client\AkamaiClientV3->purgeRequest(Array)
#17 /mnt/www/html/ncigov01live/docroot/modules/contrib/akamai/src/Plugin/Purge/Purger/AkamaiTagPurger.php(140): Drupal\akamai\AkamaiClientBase->purgeTags(Array)
#18 /mnt/www/html/ncigov01live/docroot/modules/contrib/purge/src/Plugin/Purge/Purger/PurgersService.php(505): Drupal\akamai\Plugin\Purge\Purger\AkamaiTagPurger->invalidate(Array)
#19 /mnt/www/html/ncigov01live/docroot/modules/contrib/purge/modules/purge_processor_lateruntime/src/EventSubscriber/LateRuntimeProcessor.php(104): Drupal\purge\Plugin\Purge\Purger\PurgersService->invalidate(Object(Drupal\purge_processor_lateruntime\Plugin\Purge\Processor\LateRuntimeProcessor), Array)
#20 [internal function]: Drupal\purge_processor_lateruntime\EventSubscriber\LateRuntimeProcessor->onKernelTerminate(Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...', Object(Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher))
#21 /mnt/www/html/ncigov01live/docroot/core/lib/Drupal/Component/EventDispatcher/ContainerAwareEventDispatcher.php(111): call_user_func(Array, Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...', Object(Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher))
#22 /mnt/www/html/ncigov01live/vendor/symfony/http-kernel/HttpKernel.php(115): Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch(Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...')
#23 /mnt/www/html/ncigov01live/docroot/core/lib/Drupal/Core/StackMiddleware/StackedHttpKernel.php(66): Symfony\Component\HttpKernel\HttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Symfony\Component\HttpFoundation\BinaryFileResponse))
#24 /mnt/www/html/ncigov01live/docroot/core/lib/Drupal/Core/DrupalKernel.php(715): Drupal\Core\StackMiddleware\StackedHttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Symfony\Component\HttpFoundation\BinaryFileResponse))
#25 /mnt/www/html/ncigov01live/docroot/index.php(22): Drupal\Core\DrupalKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Symfony\Component\HttpFoundation\BinaryFileResponse))
#26 {main} request_id="xxx"

Steps to reproduce

I don't exactly know the circumstances we are getting a timeout. I can see some deployments end up with the client getting 429 errors for rate limiting. This is probably from lots of cache clearing, and until Purge supports a mechanism for disabling cache clears, the large amount of cache clears will persist. It is possible that after too many requests Akamai blocks our requests as it seems like a denial of service.

However, we see this from time to time in normal content editing invalidations which should not hit any rate limits, so it could just be outages on Akamai's side.

Proposed resolution

AkamaiTagPurger and AkamaiPurger should add a try/catch block around the client request to handle any underlying exceptions.

Remaining tasks

Update purgers.

User interface changes

None

API changes

None

Data model changes

None

Issue fork akamai-3558330

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

bpizzillo created an issue. See original summary.

bpizzillo opened merge request !33

bpizzillo’s picture

bpizzillo commented
Status: Active » Needs review
gabesullice’s picture

gabesullice
Primary language English
commented
Status: Needs review » Reviewed & tested by the community

I reviewed the CR and tested this as well.

To manually test, you can force a timeout exception by manually applying this patch to src/Plugin/Client/AkamaiClientV3.php:

diff --git a/src/Plugin/Client/AkamaiClientV3.php b/src/Plugin/Client/AkamaiClientV3.php
index bf63b23..1f2a9b8 100644
--- a/src/Plugin/Client/AkamaiClientV3.php
+++ b/src/Plugin/Client/AkamaiClientV3.php
@@ -5,6 +5,7 @@ namespace Drupal\akamai\Plugin\Client;
 use Drupal\akamai\AkamaiClientBase;
 use Drupal\Core\Form\FormStateInterface;
 use GuzzleHttp\Exception\RequestException;
+use GuzzleHttp\RequestOptions;
 
 /**
  * Defines the CCUv3 client version for Akamai.
@@ -61,7 +62,10 @@ class AkamaiClientV3 extends AkamaiClientBase {
       $response = $this->client->request(
         'POST',
         "{$this->apiBaseUrl}{$this->action}/{$this->type}/{$this->domain}",
-        ['json' => $this->createPurgeBody($objects)]
+        [
+          RequestOptions::TIMEOUT => 1/1000,
+          'json' => $this->createPurgeBody($objects),
+        ],
       );
       // Note that the response has useful data that we need to record.
       // Example response body:

Then:

  1. Run drush cr && drush wd -y all
  2. Update a node via the admin UI
  3. Run drush ws --count=1
  4. Without the patch, you should see a ConnectException: cURL error
  5. Checkout the PR and install the patch
  6. Run drush cr && drush wd -y all
  7. Update a node via the admin UI
  8. Run drush ws --count=1
  9. You should see a message like purger_akamai_tag_45f74b8880: Purge Tags has encountered errors connecting to Akamai.
plach’s picture

plach
he/him
Primary language Italian
Location Venezia
commented
Status: Reviewed & tested by the community » Needs work

Code looks good but it's introducing PHPStan failures.

smustgrave made their first commit to this issue’s fork.

smustgrave opened merge request !42

  • smustgrave committed 578cd66a on 6.0.x 
    fix: #3558330 Purgers do not handle Exceptions from client

smustgrave’s picture

smustgrave commented
Version: 5.1.0 » 6.0.x-dev
Status: Needs work » Fixed

phpstan errors appeared to be resolved else where. Merged into 6.0.x and 5.1.x

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.