Automated tests that proof all features of the provider are working.
We want automated tests to mock the mistral API.
On top, it would be great to have some simple integration tests that verify compatibility with the real API.
The existing tests of php-mistral require a real Mistral API key via environment variables:
export MISTRAL_API_KEY=your-api-key
export MISTRAL_CHAT_MODEL=mistral-large-latest
The test coverage of the API clients seems in-complete, here an AI-generated analysis:
| Feature | Tested | Notes |
|---|---|---|
| Chat (stream/non-stream) | ✅ | Well covered |
| MCP Tools & Prompts | ✅ | Extensive |
| Agents | ✅ | Via MCP tests |
| Conversations | ✅ | Via MCP tests |
| Libraries CRUD | ✅ | Full coverage |
| Documents CRUD | ✅ | Full coverage |
| Embeddings | ❌ | Not tested |
| Moderation | ❌ | Not tested |
| Classifications | ❌ | Not tested |
| OCR | ❌ | Not tested |
| FIM (code completion) | ❌ | Not tested |
| File Management | ❌ | Not tested |
| List Models | ❌ | Not tested |
| Coverage | ~50% — Core chat/MCP good, embeddings/moderation/OCR missing | |
- Add test-coveraged with mocks. Evaluate and test on the best way to do it. So far AI-providers did do both guzzle middleware, but also https://mockoon.com/ to setup an actual API response. However, there is also https://docs.saloon.dev as a popular tool from the laravel universe what might be helpful. (That's used by HelgeSverre/mistral)
- Configure running the php-mistral inference tests with a public mistal API key, using its free tier. Configure a scheduled pipeline for it.
- Consider submitting upstream PRs to complete test-coverage
Let's work in sub-tasks/issues for individual steps!
Comments
Comment #2
marcus_johansson commentedComment #3
arianraeesi commentedComment #4
petar_basic commentedThe module now has 32 PHPUnit Kernel tests across 8 test files — all mocked (no real API calls).
Chat (22 tests)
- Basic chat, system roles, conversation history
- String, array, and object input formats
- Image and document file handling (base64 encoding)
- Model filtering by capability (vision, tools, JSON output)
- Streaming via iterator with event dispatch
- Response serialization to JSON-safe arrays
- Tool/function calling full lifecycle (request, response, format, JSON schema)
- Token usage tracking (input/output/total)
- API key requirement check
Embeddings (5 tests)
- String and object input
- Embedding vector output
- Max context length from API with fallback default
Moderation (4 tests)
- Safe and flagged content detection across 9 categories
- Model discovery via API capability flags (not hardcoded)
- Default model selection
Files (1 test)
- Full lifecycle: upload, signed URL, retrieve, list with filters, delete
We can add some inference tests in a follow up. Since we now have a possibility to override the mistral api host, it can also be a mocked service. Or with real mistral API key, maintainers should be able to add masked/protected variables in the module/project settings at git.drupalcode.org.
Comment #5
fagoPerfect, thank you! I think we can call this fixed, atm this seems good enough. I'll add a follow-up task for doing contract-testing with real inference API.
Comment #7
arianraeesi commented