Problem/Motivation

My goal is to get Search Api Working with a local milvus installation. In general I'm using a local lando setup for drupal.

Steps to reproduce

I've tried to setup a local Milvus Database and get it to work with milvus provider module.
I tried around with creating the milvus containers for etcd, milvus-standalone, attu and minio inside my lando.yml.
The Milvus and etcd containers are created and started, but exited shortly after, also when manually starting them again.

A second approach was to simply run a docker-compose.yml with for milvus, and then connect it to the lando_bridge_network like here at the bottom:

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.2.16
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"
      
  attu:
    container_name: attu
    image: zilliz/attu:v2.2.8
    environment:
      MILVUS_URL: milvus-standalone:19530
    ports:
      - "8000:3000"
    depends_on:
      - "standalone"


networks:
  default:
      name: lando_bridge_network
      external: true

docker inspect lando_bridge_network is at least showing all containers (appserver, database, milvus, etcd, etc.)

So in theory, they are inside the same network.

I could access attu and also create collections etc.

So I've been trying to connect Milvus via the drupal backend provider config for milvus, using the docker IP-Address of the container, localhost, "milvus-standalone" etc. together with port 19530.

Always getting "Could not connect to the server."

Running a telnet from inside the app container seems to work:

# telnet milvus-standalone 19530
Trying 172.24.0.17...
Connected to milvus-standalone.
Escape character is '^]'.

Did anyone already succeed with a local installation of milvus using lando or ddev?

Comments

netv created an issue. See original summary.

marcus_johansson’s picture

marcus_johansson commented

Ah, DDEV I can help you with.

services:
  etcd:
    container_name: ddev-${DDEV_SITENAME}-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ./milvus/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
    healthcheck:
      test: ["CMD", "etcdctl", "endpoint", "health"]
      interval: 30s
      timeout: 20s
      retries: 3
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: ${DDEV_APPROOT}

  minio:
    container_name: ddev-${DDEV_SITENAME}-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    expose:
      - "9001"
      - "9000"
    volumes:
      - ./milvus/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: ${DDEV_APPROOT}

  milvus:
    container_name: ddev-${DDEV_SITENAME}-milvus
    image: milvusdb/milvus:v2.4.1
    command: ["milvus", "run", "standalone"]
    security_opt:
    - seccomp:unconfined
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ./milvus/volumes/milvus:/var/lib/milvus
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
      interval: 30s
      start_period: 90s
      timeout: 20s
      retries: 3
    expose:
      - "19530"
      - "9091"
    depends_on:
      - "etcd"
      - "minio"
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: ${DDEV_APPROOT}

  attu:
    container_name: ddev-${DDEV_SITENAME}-attu
    image: zilliz/attu:v2.3.10
    expose:
      - "3000"
    environment:
      - MILVUS_URL=milvus:19530
      - VIRTUAL_HOST=${DDEV_SITENAME}.ddev.site
      - HTTP_EXPOSE=8521:3000
      - HTTPS_EXPOSE=8521:3000
      - SERVER_NAME=${DDEV_SITENAME}.ddev.site
    depends_on:
      - "milvus"
    labels:
      com.ddev.site-name: ${DDEV_SITENAME}
      com.ddev.approot: ${DDEV_APPROOT}
scott_euser’s picture

scott_euser commented

Ddev also has helper commands to do it for you in the VDB provider milvus documentation page (the docker compose needed is in that submodule)

scott_euser’s picture

scott_euser commented

Here I mean https://project.pages.drupalcode.org/ai/modules/vdb_providers/vdb_provid...

netv’s picture

netv commented

Thanks to both of you, worked fine :-)
Will try to adapt the config for lando aswell and post it here for refrence if successful.

marcus_johansson’s picture

marcus_johansson commented

@netv - would it be ok to close this issue?

netv’s picture

netv commented

Yes, can be closed. Thanks again for your help.

marcus_johansson’s picture

marcus_johansson commented
Status: Active » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

keiserjb’s picture

keiserjb commented

If anyone does want to get this to work with Lando. I had success with this Lando file.

name: drupal11-dev
recipe: drupal11

config:
  webroot: web
  php: '8.3'
  database: 'mysql:8.0'
  xdebug: false
  config:
    php: .lando.php.ini

services:
  appserver:
    overrides:
      environment:
        XDEBUG_MODE: 'debug,develop'
        PHP_IDE_CONFIG: "serverName=appserver"
        LANDO_HOST_IP: "host.docker.internal"
        XDEBUG_CONFIG: "remote_enable=1 remote_host=host.docker.internal"

  pma:
    type: phpmyadmin

  etcd:
    type: lando
    api: 3
    app_mount: false
    services:
      image: quay.io/coreos/etcd:v3.5.0
      command: >
        etcd -advertise-client-urls=http://127.0.0.1:2379
             -listen-client-urls http://0.0.0.0:2379
             --data-dir /etcd
      volumes:
        - ./docker/milvus/etcd:/etcd

  minio:
    type: lando
    api: 3
    app_mount: false
    services:
      image: minio/minio:RELEASE.2024-06-26T01-06-18Z
      command: minio server /minio_data
      ports:
        - "9000:9000"
        - "9001:9001"
      environment:
        MINIO_ACCESS_KEY: minioadmin
        MINIO_SECRET_KEY: minioadmin
      volumes:
        - ./docker/milvus/minio:/minio_data

  milvus:
    type: lando
    api: 3
    app_mount: false
    services:
      image: milvusdb/milvus:v2.4.5
      command: [ "milvus", "run", "standalone" ]
      ports:
        - "19530:19530"
        - "9091:9091"
      environment:
        ETCD_ENDPOINTS: etcd:2379
        MINIO_ADDRESS: minio:9000
        GRPC_PORT: 19530
      depends_on:
        - etcd
        - minio
      extra_hosts:
        - "milvus:127.0.0.1"


  attu:
    type: lando
    api: 3
    app_mount: false
    services:
      image: zilliz/attu:v2.4
      command: [ "node", "dist/src/app.js" ]
      ports:
        - "3000:3000"
      environment:
        MILVUS_URL: grpc://milvus:19530
    proxy:
      - attu-drupal11dev.lndo.site:3000

tooling:
  xon:
    service: appserver
    description: Enable Xdebug
    user: root
    cmd:
      - docker-php-ext-enable xdebug && kill -USR2 $(pgrep -o php-fpm) > /dev/null || /etc/init.d/apache2 reload
      - tput setaf 2 && echo "Xdebug On" && tput sgr 0 && echo

  xoff:
    service: appserver
    description: Disable Xdebug
    user: root
    cmd:
      - rm /usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini && kill -USR2 $(pgrep -o php-fpm) > /dev/null || /etc/init.d/apache2 reload
      - tput setaf 1 && echo "Xdebug Off" && tput sgr 0 && echo

  milvus-status:
    service: appserver
    description: Check Milvus REST health
    cmd: curl http://milvus:9091/healthz

  open-attu:
    description: Open Attu interface in browser
    service: appserver
    cmd: echo "🎉 Attu is available at http://attu.drupal11-dev.lndo.site:3000"
kristen pol’s picture

kristen pol
she/her
Primary language English
Location Santa Cruz, CA, USA
commented
Component: Providers » Miscellaneous

We are doing some issue management housekeeping and adding/removing components.

The "Providers" component will be removed because providers have been moved into their own projects, so this issue is being moved to "Miscellaneous".

See #3533272: Update AI module project components for more details.

kristen pol’s picture

kristen pol
she/her
Primary language English
Location Santa Cruz, CA, USA
commented
Project: AI (Artificial Intelligence) » Milvus VDB Provider
Version: 1.0.0-alpha7 » 1.1.x-dev
Assigned: netv » Unassigned

Issue cleanup per #3534310: Unassign closed AI issues and update issue metadata

and moving this to Milvus project

@keiserjb once an issue is closed, it's best to ask for support in the #ai Drupal Slack channel

keiserjb’s picture

keiserjb commented

Apologies, I wasn't asking for support. The original post asked about lando or ddev and this came up in my searches. My post was only for folks who might have the same search as me. Drupal Slack answers do not show up in Google.

kristen pol’s picture

kristen pol
she/her
Primary language English
Location Santa Cruz, CA, USA
commented

Gotcha! Thanks for the clarification 👍