Problem/Motivation

After drush pm:enable opentelemetry:

  [warning] Fallback OpenTelemetry endpoint http://localhost:4318 is not available. Please set the custom endpoint in the module settings. Parent exception: Export failure: Export retry limit exceeded in line 97 of /app/vendor/open-telemetry/sdk/Common/Export/Http/PsrTransport.php (previous exception message: cURL error 7: Failed to connect to localhost port 4318: Connection refused (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:4318/v1/traces). <pre>#0 /app/vendor/open-telemetry/exporter-otlp/SpanExporter.php(38): OpenTelemetry\SDK\Common\Export\Http\PsrTransport->send('\n\xB7\t\n\xB1\x06\n\e\n\thost....', NULL)
> #1 /app/vendor/open-telemetry/sdk/Trace/SpanProcessor/BatchSpanProcessor.php(247): OpenTelemetry\Contrib\Otlp\SpanExporter->export(Array)
> #2 /app/vendor/open-telemetry/sdk/Trace/SpanProcessor/BatchSpanProcessor.php(179): OpenTelemetry\SDK\Trace\SpanProcessor\BatchSpanProcessor->flush('forceFlush', NULL)
> #3 /app/vendor/open-telemetry/sdk/Trace/TracerProvider.php(57): OpenTelemetry\SDK\Trace\SpanProcessor\BatchSpanProcessor->forceFlush(NULL)
> #4 /app/web/modules/contrib/opentelemetry/src/OpentelemetryService.php(375): OpenTelemetry\SDK\Trace\TracerProvider->forceFlush()
> #5 /app/web/modules/contrib/opentelemetry/src/OpentelemetryService.php(362): Drupal\opentelemetry\OpentelemetryService->finalize()
> #6 /app/web/modules/contrib/tracer/src/EventDispatcher/TraceableEventDispatcher.php(97): Drupal\opentelemetry\OpentelemetryService->onTerminate(Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...', Object(Drupal\tracer\EventDispatcher\TraceableEventDispatcher))
> #7 /app/vendor/symfony/http-kernel/HttpKernel.php(115): Drupal\tracer\EventDispatcher\TraceableEventDispatcher->dispatch(Object(Symfony\Component\HttpKernel\Event\TerminateEvent), 'kernel.terminat...')
> #8 /app/web/core/lib/Drupal/Core/StackMiddleware/StackedHttpKernel.php(66): Symfony\Component\HttpKernel\HttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Render\HtmlResponse))
> #9 /app/web/core/lib/Drupal/Core/DrupalKernel.php(715): Drupal\Core\StackMiddleware\StackedHttpKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Render\HtmlResponse))
> #10 /app/vendor/drush/drush/src/Boot/DrupalBoot8.php(308): Drupal\Core\DrupalKernel->terminate(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Render\HtmlResponse))
> #11 [internal function]: Drush\Boot\DrupalBoot8->terminate()
> #12 {main}</pre>
 [success] Successfully enabled: opentelemetry

Also, if I visit /admin/config/development/opentelemetry:

The website encountered an unexpected error. Try again later.
Fatal error: Uncaught Symfony\Component\DependencyInjection\Exception\ServiceCircularReferenceException: Circular reference detected for service "Drupal\opentelemetry\EventSubscriber\DatabaseStatementTraceEventSubscriber", path: "Drupal\opentelemetry\EventSubscriber\DatabaseStatementTraceEventSubscriber -> opentelemetry -> opentelemetry.tracer_provider -> opentelemetry.span_processor -> opentelemetry.span.exporter -> opentelemetry.span.exporter.factory -> opentelemetry.traces.transport.factory -> opentelemetry.transport.factory.provider". in /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php:149 Stack trace: #0 /app/web/modules/contrib/tracer/src/EventDispatcher/TraceableEventDispatcher.php(88): Drupal\Component\DependencyInjection\Container->get('Drupal\\opentele...') #1 /app/web/core/lib/Drupal/Core/Database/Connection.php(2150): Drupal\tracer\EventDispatcher\TraceableEventDispatcher->dispatch(Object(Drupal\Core\Database\Event\StatementExecutionStartEvent), NULL) #2 /app/web/core/lib/Drupal/Core/Database/StatementWrapperIterator.php(109): Drupal\Core\Database\Connection->dispatchEvent(Object(Drupal\Core\Database\Event\StatementExecutionStartEvent)) #3 /app/web/core/lib/Drupal/Core/Database/Connection.php(851): Drupal\Core\Database\StatementWrapperIterator->execute(Array, Array) #4 /app/web/core/lib/Drupal/Core/Database/Schema.php(178): Drupal\Core\Database\Connection->query('SELECT 1 FROM i...', Array) #5 /app/web/core/lib/Drupal/Core/Config/DatabaseStorage.php(121): Drupal\Core\Database\Schema->tableExists('config') #6 /app/web/core/lib/Drupal/Core/Config/CachedStorage.php(95): Drupal\Core\Config\DatabaseStorage->readMultiple(Array) #7 /app/web/core/lib/Drupal/Core/Config/ConfigFactory.php(165): Drupal\Core\Config\CachedStorage->readMultiple(Array) #8 /app/web/core/lib/Drupal/Core/Config/ConfigFactory.php(104): Drupal\Core\Config\ConfigFactory->doLoadMultiple(Array, true) #9 /app/web/core/lib/Drupal/Core/Config/ConfigFactory.php(89): Drupal\Core\Config\ConfigFactory->doGet('opentelemetry.s...') #10 /app/web/modules/contrib/webprofiler/src/Config/ConfigFactoryWrapper.php(26): Drupal\Core\Config\ConfigFactory->get('opentelemetry.s...') #11 /app/web/modules/contrib/opentelemetry/src/OpentelemetryTransportFactoryProvider.php(104): Drupal\webprofiler\Config\ConfigFactoryWrapper->get('opentelemetry.s...') #12 /app/web/modules/contrib/opentelemetry/src/OpentelemetryTransportFactoryProvider.php(75): Drupal\opentelemetry\OpentelemetryTransportFactoryProvider->applyConfiguration() #13 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(261): Drupal\opentelemetry\OpentelemetryTransportFactoryProvider->__construct(Object(Drupal\webprofiler\Config\ConfigFactoryWrapper), Object(Drupal\Core\Messenger\Messenger)) #14 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(179): Drupal\Component\DependencyInjection\Container->createService(Array, 'opentelemetry.t...') #15 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(438): Drupal\Component\DependencyInjection\Container->get('opentelemetry.t...', 1) #16 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(251): Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) #17 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(179): Drupal\Component\DependencyInjection\Container->createService(Array, 'opentelemetry.t...') #18 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(438): Drupal\Component\DependencyInjection\Container->get('opentelemetry.t...', 1) #19 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(239): Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) #20 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(179): Drupal\Component\DependencyInjection\Container->createService(Array, 'opentelemetry.s...') #21 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(438): Drupal\Component\DependencyInjection\Container->get('opentelemetry.s...', 1) #22 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(251): Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) #23 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(179): Drupal\Component\DependencyInjection\Container->createService(Array, 'opentelemetry.s...') #24 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(438): Drupal\Component\DependencyInjection\Container->get('opentelemetry.s...', 1) #25 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(239): Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) #26 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(179): Drupal\Component\DependencyInjection\Container->createService(Array, 'opentelemetry.s...') #27 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(438): Drupal\Component\DependencyInjection\Container->get('opentelemetry.s...', 1) #28 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(479): Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) #29 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(239): Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) #30 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(179): Drupal\Component\DependencyInjection\Container->createService(Array, 'opentelemetry.t...') #31 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(438): Drupal\Component\DependencyInjection\Container->get('opentelemetry.t...', 1) #32 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(239): Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) #33 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(179): Drupal\Component\DependencyInjection\Container->createService(Array, 'opentelemetry') #34 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(438): Drupal\Component\DependencyInjection\Container->get('opentelemetry', 1) #35 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(239): Drupal\Component\DependencyInjection\Container->resolveServicesAndParameters(Array) #36 /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php(179): Drupal\Component\DependencyInjection\Container->createService(Array, 'Drupal\\opentele...') #37 /app/web/modules/contrib/tracer/src/EventDispatcher/TraceableEventDispatcher.php(88): Drupal\Component\DependencyInjection\Container->get('Drupal\\opentele...') #38 /app/web/core/lib/Drupal/Core/Database/Connection.php(2150): Drupal\tracer\EventDispatcher\TraceableEventDispatcher->dispatch(Object(Drupal\Core\Database\Event\StatementExecutionStartEvent), NULL) #39 /app/web/core/lib/Drupal/Core/Database/StatementWrapperIterator.php(109): Drupal\Core\Database\Connection->dispatchEvent(Object(Drupal\Core\Database\Event\StatementExecutionStartEvent)) #40 /app/web/core/lib/Drupal/Core/Database/Connection.php(851): Drupal\Core\Database\StatementWrapperIterator->execute(Array, Array) #41 /app/web/core/lib/Drupal/Core/Database/Query/Select.php(524): Drupal\Core\Database\Connection->query('SELECT 1 AS "ex...', Array, Array) #42 /app/web/core/lib/Drupal/Core/Database/Query/Merge.php(379): Drupal\Core\Database\Query\Select->execute() #43 /app/web/core/lib/Drupal/Core/Session/SessionHandler.php(95): Drupal\Core\Database\Query\Merge->execute() #44 /app/web/core/lib/Drupal/Core/Session/SessionHandler.php(97): Drupal\Core\Session\SessionHandler->Drupal\Core\Session\{closure}() #45 /app/web/core/lib/Drupal/Core/Session/WriteSafeSessionHandler.php(95): Drupal\Core\Session\SessionHandler->write(Object(SensitiveParameterValue), '_sf2_attributes...') #46 /app/vendor/symfony/http-foundation/Session/Storage/Proxy/SessionHandlerProxy.php(54): Drupal\Core\Session\WriteSafeSessionHandler->write('vq6clh8AjpAnRz1...', '_sf2_attributes...') #47 [internal function]: Symfony\Component\HttpFoundation\Session\Storage\Proxy\SessionHandlerProxy->write(Object(SensitiveParameterValue), '_sf2_attributes...') #48 [internal function]: session_write_close() #49 {main} thrown in /app/web/core/lib/Drupal/Component/DependencyInjection/Container.php on line 149

Running PHP 8.3.14, Drupal 10.4.1, module_opentelemetry 1.1.0

Issue fork opentelemetry-3498114

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

jonathan_hunt created an issue. See original summary.

nidhish’s picture

nidhish commented
Assigned: Unassigned » nidhish
jonathan_hunt’s picture

jonathan_hunt commented
Assigned: nidhish » Unassigned

The Fatal error: Uncaught Symfony\Component\DependencyInjection\Exception\ServiceCircularReferenceException: Circular reference detected for service is mentioned in the code at https://git.drupalcode.org/project/opentelemetry/-/blob/1.0.x/src/EventS...

murz’s picture

murz
Primary language English
Location Yerevan, Armenia
commented

Thanks for reporting! Will try to fix this.

The message:
[warning] Fallback OpenTelemetry endpoint http://localhost:4318 is not available. Please set the custom endpoint in the module settings. Parent exception: Export failure: Export retry limit exceeded
is expected, it just warns you that the endpoint is not available.

But about the exception - I can't reproduce the problem on my side to debug and investigate.

Please share the Drupal version and installed modules on your side, and any other non-standard environment settings.

jonathan_hunt’s picture

jonathan_hunt commented

I had https://www.drupal.org/project/webprofiler installed and that relies upon https://www.drupal.org/project/tracer. If I uninstall both then /admin/config/development/opentelemetry serves ok...

berdir’s picture

berdir
Primary language German
Location Switzerland
commented

> is expected, it just warns you that the endpoint is not available.

This is similar to issues I opened, which changed it from an error to a warning, but I think that's still problematic.

I think there really needs to be a complete off switch without having to uninstall the modules, for example for non-production/local development environments, CI and also initial install.

There is the disable checkbox, but currently it's only respected for traces (without explaining that) and not logs.

Drupal modules can include a link to their configuration site which is shown after install now by default in drush, there could be a runtime requirements hook to show an error if the connection fails.

berdir’s picture

berdir
Primary language German
Location Switzerland
commented

How about this idea as a compromise?

Require an explicit configuration of the endpoint, if the field is empty then skip tracing/logging/whatever, but ship it with localhost as default configuration? 

if ($this->settings->get(OpentelemetryService::SETTING_ENDPOINT))  {
      $this->logger = $this->loggerProvider->getLogger(
        $this->settings->get(OpentelemetryService::SETTING_SERVICE_NAME) ?? OpentelemetryService::SERVICE_NAME_FALLBACK
      );
    }
    else {
      $this->logger = NoopLogger::getInstance();
    }

There might be better ways to set that up, but it wasn't obvious to me.

And make the disable checkbox in the UI clearer: "Disable tracing"

Or, make the disable checkbox apply to everything and then check for that instead of the endpoint. But I actually see use cases for having logging enabled but not tracing. Especially since I won't be able to have a local opentelemetry collector to batch data together, I'm still unsure how much data I really want to collect and process.

berdir opened merge request !61

berdir’s picture

berdir
Primary language German
Location Switzerland
commented

Created a merge requests, doesn't quite fit the actual issue, can also move it somewhere.

berdir closed merge request !61

berdir’s picture

berdir
Primary language German
Location Switzerland
commented

I closed my merge request, this gets more complicated with #3505594: Update log structure to provide body as string as well as standardized attributes because we no longer have the ConfigFactory in OpenTelemetryLogs. What we did in our project now is alter the logger_provider service with this:

class ConditionalLoggerProvider extends LoggerProvider {

  /**
   * The config factory.
   */
  protected ConfigFactoryInterface $configFactory;

  public function setConfigFactory(ConfigFactoryInterface $configFactory): void {
    $this->configFactory = $configFactory;
  }

  /**
   * {@inheritdoc}
   */
  public function getLogger(string $name, ?string $version = NULL, ?string $schemaUrl = NULL, iterable $attributes = []): LoggerInterface {
    if ($this->configFactory->get('opentelemetry.settings')->get('endpoint')) {
      return parent::getLogger($name, $version, $schemaUrl, $attributes);
    }
    else {
      return NoopLogger::getInstance();
    }
  }

}

This works for us, I still think that something like this should be the default behavior though.