Motivation

With #3566649: Add test coverage we have solid test-coverage with mocked inference API from mistral. It would be great to some foundational contract/integration tests ensuring everything works well with the real inference API as well.

Proposed resolution

  • Use free trial key from mistral, limited runs
  • Run test on weekly schedule or similar
  • Configure key via protected secret at gitlab
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

fago created an issue. See original summary.

fago’s picture

Issue summary: View changes

nickolaj made their first commit to this issue’s fork.

nickolaj’s picture

Status: Active » Needs review

Adds contract tests for chat, embeddings, and moderation operations using the real Mistral inference API. Tests are grouped under `ai_provider_mistral_contract`, excluded from regular CI runs, and configured to run via a weekly scheduled pipeline with the `MISTRAL_API_KEY` CI/CD variable.

mattlc’s picture

Because of Drupal\ai\Base\AiProviderClientBase::loadApiKey() method we cannot set the key value directly into the setting, this have to refer to a Key entity.

With the modifications bellow, I was able to run the tests without the "Could not load the %s API key, please check your environment settings or your setup key" error.

In MistralContractTestBase, these modifications make the tests pass using the MISTRAL_API_KEY env variable :

  /**
   * {@inheritdoc}
   */
  protected function setUp(): void {
    $api_key = getenv('MISTRAL_API_KEY');
    if (empty($api_key)) {
      $this->markTestSkipped('MISTRAL_API_KEY environment variable is not set.');
    }

    parent::setUp();

    $key = Key::create([
      'id' => 'mistral_api_key',
      'label' => 'Mistral api key',
      'key_type' => 'authentication',
      'key_provider' => 'env',
      'key_provider_settings' => [
        'env_variable' => 'MISTRAL_API_KEY'
      ]
    ]);
    $key->save();

    // Store the real API key in configuration.
    $this->container->get('config.factory')
      ->getEditable('ai_provider_mistral.settings')
      ->set('api_key', 'mistral_api_key')
      ->save();

    /** @var \Drupal\ai\AiProviderPluginManager $provider_manager */
    $provider_manager = $this->container->get('ai.provider');
    $this->provider = $provider_manager->createInstance('mistral');
  }

There are 2 remaining failing tests:

  • ChatContractTest::testStreamingChat => Failed asserting that a string is not empty.
  • StreamingTest::testStreamingChat => Failed asserting that two arrays are equal.

It seams that the streaming chat is broken (empty chunks returned).