Long Haul

Long Haul hands slow jobs that can't be split up (database dumps, database imports and search reindexes) to longhaul-worker, a small companion service written in Go. Drupal starts a job with one short HTTP request, the worker runs it in a goroutine, and an admin page shows its progress until it finishes. No PHP request, browser tab, proxy timeout or max_execution_time has to last as long as the job.

Use it when a job can't be cut into Batch API steps. A multi-gigabyte mysql import is one continuous stream into one process, and a full Solr reindex boots Drupal again for every batch. If your job splits into small pieces and finishes in a minute, the Batch API or a queue worker is less to run, and you should use those instead.

The module is deliberately thin. It never does the work itself. It is a client for the worker's three-endpoint HTTP API. The design is explained in detail in Let PHP start it, let Go finish it: long-running Drupal jobs as goroutines.

Features

• Three built-in jobs. Dump the database (mysqldump streamed through gzip into a .sql.gz file), import a dump (the reverse: file, gunzip, mysql), and rebuild the search index (drush search-api:index, run and supervised by the worker).
• Short requests only. The worker answers 202 Accepted before it starts working, so every call from Drupal finishes in milliseconds under a 5-second timeout. No PHP worker is tied up while the job runs.
• Live progress. After you start a job, the admin page polls every two seconds and shows the job's state and megabytes processed. Polling stops when the job is done, has failed or was cancelled.
• Flat memory use. Dumps and imports are streamed, so the worker uses the same memory for a 50 MB database as for a 50 GB one. A failed dump is deleted, never left half-written.
• Lanes prevent collisions. Dump and import share a database lane, so you can never import over a dump that is still being written. A reindex runs in its own lane alongside them. Starting a job in a busy lane is refused, and the form names the job that is running. The rule is enforced in the worker behind a mutex, so it holds however many requests arrive at once.
• Cancellation. The worker cancels a running job by killing its child process. The WorkerClient service exposes start(), status() and cancel() for use in your own code.
• Audit trail. Every request carries X-Longhaul-Actor: drupal:<username>, and the worker logs who started each job.
• Secrets stay out of config. The worker URL and token live in settings.php, so copying a database to another environment never carries them along.
• Cache-safe status. The browser never talks to the worker. Status is relayed through a Drupal route that answers private, no-store, so Varnish or a CDN can never serve one admin's job status to another.
• A small worker. One main.go with no dependencies outside the Go standard library, built into a single static binary. It uses bearer-token authentication with a constant-time comparison, a three-hour timeout per job, and graceful shutdown on SIGTERM.

Post-Installation

1. Build and run longhaul-worker. Set LONGHAUL_TOKEN (required) and, if the defaults don't fit, LONGHAUL_DUMP_DIR (default /data/dumps), LONGHAUL_MYCNF (default /etc/longhaul/my.cnf), LONGHAUL_DB (default drupal) and LONGHAUL_DRUPAL_ROOT (default /app). The worker listens on port 8080.
2. Give the worker what its jobs need. Dumps and imports need the mysql and mysqldump clients, a my.cnf with database credentials (mounted from a secret) and a volume for dump files. Reindexing needs Drush and your codebase. The easiest option is a sidecar container in the Drupal pod, built from the same image with the Go binary copied in.
3. Add the worker's address and token to settings.php:
   

   $settings['longhaul_url'] = 'http://longhaul-worker:8080';
   $settings['longhaul_token'] = getenv('LONGHAUL_TOKEN');

4. Grant Run long-running jobs only to trusted roles. The permission is marked restrict access because an import replaces the site's database.
5. Go to /admin/reports/longhaul, choose a job and click Start. For an import, enter the dump's file name, for example 4ef17a45299aefce.sql.gz. Only .sql.gz files in the dump directory are accepted.

Keep the worker off the public internet. Only Drupal should be able to reach it. On Kubernetes, a NetworkPolicy makes that explicit.

You can test the worker with curl before Drupal is involved:

curl -s -H "Authorization: Bearer $LONGHAUL_TOKEN" -d kind=dump localhost:8080/jobs
curl -s -H "Authorization: Bearer $LONGHAUL_TOKEN" localhost:8080/jobs/<id>

Limitations

• The worker keeps jobs in memory. Restarting it forgets them, and SIGTERM cancels any job that is running. That is fine for a weekly dump. If a job must survive a restart, it needs persistent state (see Stagecoach under Similar projects).
• The admin page shows the job you just started. It has no job history list and no cancel button yet.
• Dump and import support MySQL and MariaDB only.
• You need somewhere to run a long-lived process, so Long Haul is not an option on shared hosting.

Additional Requirements

• Drupal 10.3 or later, or Drupal 11.
• longhaul-worker, available at [WORKER REPO URL]. Building it requires Go 1.22 or later, because it uses net/http's method and path routing patterns.
• MySQL or MariaDB, with the mysql and mysqldump clients available wherever the worker runs.
• For the reindex job: Search API and Drush, available to the worker.

Recommended modules/libraries

• Search API with a backend such as Search API Solr, for the reindex job.

Similar projects

• Drupal core's Batch API and Queue API are the right tools when work splits into small chunks. Long Haul is for work that can't be split, or that must not depend on a browser tab or on cron.
• Backup and Migrate offers scheduled backups, multiple destinations and restores, all run inside Drupal. Long Haul does much less, but it streams mysqldump in a separate process, so dump size isn't bound by PHP's limits.
• Advanced Queue provides a persistent job queue with job types, states and retries, processed by cron or Drush. It's a good fit when jobs can run in PHP. Long Haul moves them out of PHP's lifecycle entirely.
• Stagecoach uses the same pattern for production deployments. Deploys, snapshots and copy-downs run as Kubernetes Jobs in a Go service with SQLite-backed history, so they survive restarts. Long Haul is a minimal, generic version of that design.

Supporting this Module

Bug reports, feature requests and merge requests are welcome in the issue queue.

Community Documentation

• Let PHP start it, let Go finish it: the full write-up, including all of the module and worker code.
• longhaul-worker source
• The module's README.md